Don’t Look Up: The Cost of Attention to Stimulus Phrases in Mobile Text Entry Evaluations

Transcription tasks have been long used as the de-facto evaluation method in mobile text entry research. Evaluations use memorable phrase sets, in order to prevent participants from devoting more attention to the stimulus phrase than the bare minimum. We present evidence from an eye-tracking study, demonstrating that the attention devoted to the stimulus phrase is much higher than might be expected. In fact, attention to the stimulus phrase takes up almost 50% of participant attention spent outside the keyboard area, and overall 25% of participant attention throughout any single transcription task. We explore a modification to the transcription task aimed at reducing this level of visual attention, without finding any statistically significant differences. These findings raise important questions on the continued use of the transcription task as the mainstream evaluation method for mobile text entry research.