Multimodal desktop interaction: The face - object - gesture - voice example

Abstract

This paper presents a natural user interface system based on multimodal human computer interaction, which operates as an intermediate module between the user and the operating system. The aim of this work is to demonstrate a multimodal system which gives users the ability to interact with desktop applications using face, objects, voice and gestures. These human behaviors constitute the input qualifiers to the system. Microsoft Kinect multi-sensor was utilized as input device in order to succeed the natural user interaction, mainly due to the multimodal capabilities offered by this device. We demonstrate scenarios which contain all the functions and capabilities of our system from the perspective of natural user interaction.