Integrating User Gaze with Verbal Instruction to Reliably Estimate Robotic Task Parameters in a Human-Robot Collaborative Environment

Abstract

As robots become more ubiquitous in our daily life, it has become very important to extract task and environmental information through more natural, meaningful, and easy-to-use interaction interfaces. Not only this helps the user to adapt to (thus trust) a robot in a collaborative environment, it can supplement the core sensory information, helping the robot make reliable decisions. This paper presents a framework that combines two natural interaction interfaces: speech and gaze to reliably infer the object of interest and the robotic task parameters. The gaze estimation module utilizes pre-defined 3D facial points and matches them to a set of extracted estimated 3D facial landmarks of the users from 2D images to infer the gaze direction. Subsequently, the verbal instructions are passed through a deep learning model to extract the information relevant to a robotic task. These extracted task parameters from verbal instructions along with the estimated gaze directions are combined to detect and/or disambiguate objects in the scene to generate the final task configurations. The proposed framework shows very promising results in integrating the relevant task parameters for the intended robotic tasks in different real-world interaction scenarios.