Relevant Perception Modalities for Flexible Human-Robot Teams

Robust and reliable perception plays an important role when humans engage into cooperation with robots in industrial or household settings. Various explicit and implicit communication modalities and perception methods can be used to recognize expressed intentions. Depending on the modality, different sensors, areas of observation, and perception methods need to be utilized. More modalities increase the complexity and costs of the setup. We consider the scenario of a cooperative task in a potentially noisy environment, where verbal communication is hardly feasible. Our goal is to investigate the importance of different, non-verbal communication modalities for intention recognition. To this end, we build upon an established benchmark study for human cooperation and investigate which input modalities contribute most towards recognizing the expressed intention. To measure the detection rate, we conducted a second study. Participants had to predict actions based on a stream of symbolic input data. Findings confirm the existence of a common gesture dictionary and the importance of hand tracking for action prediction when the number of feasible actions increases. The contribution of this work is a usage ranking of gestures and a comparison of input modalities to improve prediction capabilities in human-robot cooperation.