A multi-modal intelligent user interface for supervisory control of unmanned platforms

Abstract

Typical human-robot interaction (HRI) is through tele-operation or point-and-click interfaces that require extensive training to become proficient and require the user's complete attention to operate. For unmanned platforms to reach their full potential, users must be able to exert supervisory control over those platforms. This requires more effective means of communication in both directions, including high-level commands given to the vehicle and meaningful feedback to the user. Our aim is to reduce the training requirements and workload needed to interact with unmanned systems effectively and to raise the level of user interaction with these systems so that supervisory control is possible. In this paper we describe an intelligent user interface, called the Smart Interaction Device (SID) that facilitates a dialogue between the user and the unmanned platform. SID works with the user to understand the user's intent, including asking any clarification questions. Once an understanding is established, SID translates that intent into the language of the platform. SID also monitors the platform's progress in order to give feedback to the user about status or problems that arise. We have incorporated multiple input modalities, including speech, gesture, and sketch as natural ways for a user to communicate with unmanned platforms. SID also provides multiple modes of feedback, including graphics, video and speech. We describe SID's architecture and some examples of its application in different domains.