Layered Evaluation of a Personalized Interaction Approach

Evaluation of personalized systems is a complicated endeavor. First, evaluation goals, methods and criteria are manifold and have to be carefully selected to fit the actual application scenario and the scope of the evaluated system. Second, it is considerably harder to locate the source of problems, compared to non-adaptive systems where problems most often reside on the UI level. Thus, in the past, a layered evaluation approach for personalized systems has been proposed that distinguishes between five layers that can theoretically all be the source of problems (e.g., collection of input data or adaptation decision). This paper outlines a use case related to personalized interaction comprising i) modeling a user's interaction abilities, ii) recommending interaction methods and devices that fit the user's individual needs, and iii) personalized system behavior and reaction to user input. Next, the paper describes experiences with an evaluation process using the layered evaluation framework.