Changing Analysts’ Tunes: The surprising impact of a new instrument for usability inspection method assessment

Cockton, Gilbert, Woolrych, Alan, Hall, Lynne and Hindmarch, Mark (2004) Changing Analysts’ Tunes: The surprising impact of a new instrument for usability inspection method assessment. In: People and Computers XVII — Designing for Society. Springer, London, pp. 145-161. ISBN 9781852337667

Full text not available from this repository. (Request a copy)

Abstract

We describe the impact on analyst performance of an extended problem report format. Previous studies have shown that Heuristic Evaluation can only find a high proportion of actual problems (thoroughness) if multiple analysts are used. However, adding analysts can result in a high proportion of false positives (low validity). We report surprising interim results from a large study that is exploring the DARe model for evaluation method effectiveness. The DARe model relates the effectiveness of an evaluation method to evaluators’ command of discovery and analysis resources. Previous work has shown that Heuristic Evaluation poorly supports problem discovery and analysis: heuristics tend to be inappropriately applied to problem predictions. We developed an extended problem report format to let us study analyst decision making during usability inspection. Our focus was on the quality of insights into analyst behaviour delivered by this extended report format. However, our first use of this format revealed unexpected improvements in validity (false positive reduction) and appropriate heuristic application. We argue that the format has unexpectedly led to more care and caution in problem discovery and elimination, and in heuristic application. Evaluation performance can thus be improved by indirectly ‘fixing the analyst’ via generic fixes to inspection methods. In addition, we provide the first direct evidence of how evaluators use separate discovery and analysis resources during usability inspection.

Item Type:	Book Section
Additional Information:	John Long Award for Best Paper. Proceedings of HCI 2003.
Subjects:	G400 Computer Science
Department:	Faculties > Arts, Design and Social Sciences > Design
Depositing User:	Ay Okpokam
Date Deposited:	19 Jan 2016 11:45
Last Modified:	12 Oct 2019 19:40
URI:	http://nrl.northumbria.ac.uk/id/eprint/25533

Downloads