An Evaluation Framework for Assessing and Optimizing Multimodal Fusion Engines Performance

Abstract

The current development of interactive systems is shifting its focus into adding new features and capabilities, encompassing for example, new input devices and ways of interacting. Some applications make use of different modalities for both input and output, which adds great complexity to such systems. Due to a possible high number of input modalities and devices available, the task of combining all the information conveyed by users from these sources, becomes critical and troublesome. This process is commonly referred to, as multi modal fusion, and is performed by fusion engines, components of multi modal systems, that have the purpose of receiving multiple streams of input, combine them, and reaching an interpretation of user intent. Evaluating the efficiency of this process is a task that requires a considerate amount of effort, due to all the variables involved. In this paper we present an evaluation framework and methodology aimed towards the assessment and optimization of fusion engines performance. We begin by correlating our work to project GUIDE, explaining how fusion is achieved and how information about user and context are used for adaptation. Subsequent sections discuss the proposed framework, and how it enables developers to assess and improve their components performance. The article concludes by showing some results that demonstrate the benefits of using such an approach and the impact it can have on the development of fusion engines.