An initial exploration of conversational errors as a novel method for evaluating virtual human experiences

Abstract

We present a new method for evaluating user experience in interactions with virtual humans (VHs). We code the conversational errors made by the VH. These errors, in addition to the duration of the interaction and the numbers of statements made by the participant and the VH, provide objective, quantitative data about the virtual social interaction. We applied this method to a set of previously collected interactions between medical students and VH patients and present preliminary results. The error metrics do not correlate with traditional measures of the quality of a virtual experience, e.g. presence and copresence questionnaires. The error metrics were significantly correlated with scores on the Maastricht Assessment of Simulated Patients (MaSP), a scenario-appropriate measure of simulation quality, suggesting further investigation is warranted.