Synchronization of speech and hand gestures during multimodal human-computer interaction

Abstract

In this paper, we describe an experiment that studies temporal synchronization between speech (Japanese) and hand pointing gestures. Gesture (G) is shown to be synchronized with either the nominal or deictic (“this”, “that”, ‘here”, etc.) expression of a phrase. It is also shown that G is predictable in the [-200 ms, 400 ms] interval around the beginning of its related expression. The use of such a quantitative model of natural speech and gesture integration (in the multimodal interface and the speech recognition system), is also discussed