On the reliability of usability testing

Abstract

Six professional usability testing teams conducted a usability test on an early prototype of a dialog box. Altogether, they identified 36 usability problems. No problem was detected by every team, 2 were found by five teams, 4 by four teams, 7 by three teams, 7 by two teams, and 18 problems were identified by one team only. There was more agreement among teams in this study compared to a previous study [1] and there was more agreement among the teams on severe vs. minor problems. Implications for the cooperation between usability testers and their clients are discussed.