Simultaneous Integration of Multimodal Interfaces for Generating Structured and Reliable Robotic Task Configurations

Abstract

This paper presents a framework that simultaneously integrates multiple input interfaces and extracts task parameters suitable for task execution in a human-robot collaborative environment. We used pointing gestures and natural language instruction as inputs as they provide the most natural interaction interfaces for humans. In the proposed method, the pointing gesture type and the pointing direction are estimated from RGB images, and the object being pointed at is inferred from the prior gesture information and the objects detected in the scene. Subsequently, the verbal command is parsed to extract task action, the object of interest along with its attributes and position in the 2D image frame. This extracted information from gesture recognition and verbal command is used to form task configurations for the desired human-robot collaborative tasks as well as to help resolve any uncertain or missing task parameters. The proposed framework shows very promising results in identifying the relevant task parameters for the intended robotic tasks in different real-world interaction scenarios.