User-Centered Multimodal Interaction Graph for Design Reviews

Abstract

This work presents a novel approach to author multimodal interaction dialogue of a VR system according to each users specific preferences. We will show how modalities can be bound together via a bidirectional graph in an authoring tool to allow the specification of application-specific domain commands without hardwiring them to the application. As a result we provide a persistent definition of the used modalities outside the application. This is done through the adoption of a so-called ";interaction graph"; whose nodes and edges represent the dialogue of the user with the system. The application then identifies interaction patterns by matching the path within the graph that represents the actions the user wants to perform within the application.