User modeling and knowledge-based support: on the evaluation of improved system design

Abstract

Various approaches to user modeling are presented and the possibility for verification is assessed. In a larger context, this leads to the issue of evaluating and validating combined human-machine systems in the presence of model-based and expert-system-based operator support. With such support, detailed measures and analyses of the overall system performance become more important. An example of a project that deals with a multi-expert-system supported graphical dialogue environment (GRADIENT) illustrates some of the aspects of designing experiments and other suitable approaches for assessing the system's impact on the human operator's behavior and performance. The general guideline is to plan evaluation bottom up, combining many specific or restricted aspects in order to gain a comprehensive understanding of the overall systems. It is suggested that, even with full use of existing methodologies, the evaluation of complex operator support systems may fail, and, therefore, directions for research are proposed that appear promising.<>