Is usability testing valid with prototypes where clickable hotspots are highlighted upon misclick?

Abstract

In user experience design, prototypes are an indispensable tool for early diagnosis of usability issues. Designing usability testing to accommodate a prototype’s limited interactivity is essential to obtain relevant participant feedback. Hotspot Highlighting is a technique employed by all prominent prototyping tools to allow usability testers to see which areas of the prototype are clickable. The current body of knowledge lacks definite answers on how highlighting impacts usability testing results, compared to scenarios where participants complete tasks fully on their own. Can studies be treated the same, regardless of whether Hotspot Highlighting is enabled? What are the recommendations for how and when Hotspot Highlighting can or should be used? To investigate, we conduct a between-subjects experiment with 80 participants and 240 task completions in which we compare user behavior depending on the presence of Hotspot Highlighting. Its results indicate that Hotspot Highlighting can affect participant behavior before and after a highlight is displayed, leading to potentially different usability findings if left unaccounted for. The guidance of highlights changes the targets of clicks and encourages cognitively efficient finding of solutions by intentionally triggering the highlights. Considering the potential of Hotspot Highlighting to facilitate the usability testing of prototypes with limited interactivity, we discuss potential adaptations of the technique that address its current issues for more methodologically sound usability evaluation.