Multimodal interaction architecture applied to navigation in maps

Abstract

This paper presents a multimodal interaction architecture that is proposed as the interaction/control component of new or existing computer applications, particularly for 2D/3D visual computing applications. The architecture aims to encourage multimodality in interaction between the user and the computer application in order to achieve a more natural interaction, easier and user friendly. The presented architecture integrates specialized interaction modules in different modalities (hand gestures and voice initially) operating simultaneously. The information resulting from these modules is processed by a multimodal integration module to detect simultaneous actions/commands. The architecture was integrated with Bing-Maps application to allow navigation in maps through voice and gestures. Some results of an initial evaluation using the current prototype are presented where we get information regarding its functionality and the resulting user experience.