Cueing Retrospective Verbal Reports in Usability Testing Through Eye-Movement Replay

An experimental validation is presented of a novel method for usability testing that entails the playback of dynamic eyetracking data to cue the elicitation of retrospective verbal reports. Participants in our study produced: (1) think-aloud reports during an online search task, and (2) retrospective reports during another online search task, with reports being cued by the playback of either the screen capture of events or the participant’s own eye-movements. Task-completion times and response rates were recorded for all reporting methods. Fewer participants completed the search task whilst thinking aloud, indicating the reactivity of this technique. Verbal transcripts were coded for instances of usability problems. The eye-cued method identified more usability problems than the think-aloud or screen-cued methods. A significant interaction between search engine type and retrospective cue type suggests that the value of the eye-cue method for eliciting usability problems may be greatest with more complex search environments. Our results demonstrate that when cued appropriately, retrospective reports may be less reactive and more informative than other verbalisation techniques.