Programming the multimodal interface

Abstract

Fundamental problems will confront those who wish to take full advantage of the power of tomorrow`s multimodal environments. We argue that our recently introduced concept of meta-widget, when embedded within a high level, networked user interface server, can support the effective implementation of complex multimedia applications. We develop algorithms which enable a multimodal system to select the ``best`` combination of representations for the various ``information packets`` in a display at any moment. If no acceptable - combination of available representations can be found, strategies are provided for creating a new and useful, if not beautiful, representation to resolve the impasse. A running example is provided to motivate and clarify the discussion.