Empirical Evaluation of User Modeling Systems

Abstract

This tutorial will introduce User Modeling (UM) researchers to the techniques of empirical evaluation of user modeling systems. No background in statistics is required. The target audience is UM researchers, especially students, who have a background in computer science or some other field that does not normally include designing and running human-subject experiments. Topics include designing experiments (choosing independent/dependent variables, covariant and nuisance variables, between vs. within subjects designs, factorial designs, estimating sensitivity, layered evaluation), running experiments (recruiting participants, controlling the environment, recording data), data analysis (statistical tests, ANOVA, checking assumptions of statistical methods, multiple testing correction, explained variance), and common surveys/tests for gathering covariate data.