A German-Chinese speech-gesture behavioural corpus of device control in a smart home

Abstract

This paper reports on empirical studies on the use of speech and gesture interaction for device control in the Bremen Ambient Assisted Living Lab (BAALL) at the German Research Center for Artificial Intelligence at the University of Bremen. Our goal is to create and evaluate a multimodal corpus of device control (turn on lights, open sliding door, etc.). A Wizard-of-Oz user study was conducted in this lab with German and Chinese participants in order to carry out a comparative analysis of cross-lingual spatial spoken and gesture commands and report on cultural differences. The long-term goal is to design and develop a speech and gesture combining multimodal interactive system, while considering relevant behavioural aspects, such as cultural differences presented in this paper.