Novice evaluators' behavior when consolidating usability problems individually or collaboratively

Abstract

An important, but resource demanding step in analyzing observations from usability evaluations is to consolidate usability problems (UPs) that were identified by several evaluators into one master list. An open question is whether consolidating UPs in pairs is cost-effective. A within-subject study examined if evaluators merge UPs differently when working in pairs than individually and what motivates their decisions. Eight novice evaluators took part. The number of discarded, retained and merged UPs, evaluators' confidence and severity of UPs in the two settings were measured. The results showed that UPs merged or discarded in the collaborative setting would rather be retained in the individual setting. Participants increased confidence and UP severity in the collaborative setting but decreased UP severity and confidence in the individual setting.