A Question of Fidelity: Comparing Different User Testing Methods for Evaluating In-Car Prompts

Abstract

User studies are a major component in any user-centered design process. Testing methods thereby vary tremendously regarding the organizational, financial, and timely effort needed to conduct them. Driving simulator studies generally are the method of choice when dialogs need to be validated for in-car settings. These studies are highly time- and cost-consuming though. Online crowdsourcing studies can be an alternative as they allow for quick results and large sample sizes while at the same time being time- and cost-efficient. Still, voice user interface designers argue for a lack of applicability to concrete use cases. This is especially true for speech dialog systems in an in-car context where users experience voice as a secondary task with the primary task being driving. To compare the validity of different user testing methods, study participants in a between-subjects study design evaluated proactive in-car prompts presented a) in an online crowdsourcing study in text form, b) in an online crowdsourcing study via audio, and c) in a driving simulator. Prompt evaluations did not differ significantly between conditions a) and c) but diverged for condition b). Findings are explained by drawing from the Elaboration Likelihood Model and used to answer the question of how to efficiently validate in-car prompts.