Using simulated students to evaluate an adaptive testing system

Abstract

This paper describes a strategy for evaluating an assessor algorithm through the use of simulated students. The assessor is an adaptive testing strategy elicited from a human expert. The assessor claims an accurate assessment of the student's domain knowledge with a minimum number of questions presented to the student. An evaluation of the assessor is carried out to test the claim. Three types of students are simulated-one who knows all the relevant skills in the subject domain, one who knows a subset of the relevant skills, and one who has one or more malrules. The performance of the assessor in predicting the mastery of skills of each student is analysed. Future work involves the evaluation of its accuracy in the assessment of other types of students, such as students with lucky guesses or careless slips.