Pilot Studies: When and how to conduct them when conducting user studies

Abstract

Evaluating an interactive system with users can be costly and timeconsuming. As a result, it can be extremely frustrating for evaluators whenever they are not able to use any of the data collected from the user evaluation of a system because of problems with how the study was designed or executed. In such instances, evaluators must learn from their mistakes and rerun the study a second time. However, the time, effort, and resources that went into completing the study the first time is still wasted. This problem can be minimized by first conducting a pilot study -- a small-scale version of the actual study -- prior to the final study, to identify, remove, and avoid any potential problems.