Spoken conversational search: Evaluating the effect of system clarifications on user experience through Wizard-of-Oz study

Abstract

Prior research in human–computer interaction suggests that system-level clarifications are necessary for understanding user intent and communicating effectively with the user. Such clarifications or explanations could contain the system's abstract knowledge of the search or a functional description of the search process (queries and information sources employed). While these interactions may aid the user and the agent in better understanding each other, very few studies have explored the influence of such clarifications on the users' search experience. This research examines whether and how system-level clarifications (or explanations) affect the user experience when searching through spoken dialogues. We analyzed user satisfaction and preferences in systems with and without explicit clarifications in a within-subjects Wizard-of-Oz user study. We recruited 25 participants and collected user–system interaction data for 50 search sessions. The user feedback was collected using pre- and post-task surveys and exit interviews. Statistical and qualitative analysis of user responses yielded some interesting findings. While Wilcoxon Signed Rank Test found that using explicit system-level clarifications had no positive influence on the user's search experience, the overall search experience degraded with system clarifications (Z = −2.066, p = 0.04). The user interview data provided valuable insights into how and when clarifications should be offered to the user.