Wizards in the Middle: An Approach to Comparing Humans and Robots*

Abstract

While Wizard-of-Oz (WOz) techniques are frequently used to supplement a machine’s abilities, extending this approach to human entities can increase experimental control in studies comparing evaluations of humans and machines in the same role. This article describes the design, implementation, and use of a WOz system for facilitating controlled verbal interactions between children and robot or human interviewers. A collaborative interface allows multiple remote wizards to combine participant responses with interaction-specific goals in order to direct a robot or human interviewer’s behavior in a consistent manner. While robot interviewers are controlled directly, human interviewers receive direction through a tablet device or via a projection system concealed from participants. In addition to the system’s technical design, we describe the division of responsibilities between wizards and insights from using the system across three extensive interview studies to facilitate a total of 217 interactions with children.