Adaptive multimodal interaction with industrial robot

Abstract

This paper reports a spoken natural language dialogue system that manages the interaction between the user and the industrial robot ABB IRB 140. To the extent that the dialogue system is multimodal, it uses three communication modalities: (i) spoken language (automatic speech recognition and text-to-speech synthesis), (ii) visual recognition of the figures and determination of their positions, and (iii) typed text. To the extent that the dialogue system is adaptive, it takes the verbal and spatial contexts into account in order to adapt its dialogue behavior and to process spontaneously formulated user commands of different syntactic forms without explicit syntactic expectations. The industrial robot is slightly modified and enabled to manipulate over graphical figures, following the instructions of the dialogue system.