Modular approach of multimodal integration in a virtual environment

Abstract

We present a novel modular approach to integrating multiple input/output (I/O) modes in a virtual environment that imitate natural, intuitive and effective human interaction behavior. The I/O modes used in this research are spatial tracking of both hands, finger gesture recognition, head/body spatial tracking, voice recognition (discrete recognition for simple commands, and continuous recognition for natural language input), immersive stereo display and synthesized speech output. Intuitive natural interaction is achieved through several stages: identifying all the tasks that need to be performed, grouping similar tasks and assigning them to a particular mode such that it imitates the physical world. This modular approach allows inclusion and removal of additional input and output modes as well as additional users. We described this multimodal interaction paradigm by applying it to a real world application: visualizing, modeling and fitting protein molecular structures in an immersive virtual environment.