Ordered Partition Model for Confidence Marking Modeling.

Abstract

Confidence marking is increasingly used in multiple choice testing situations, but when the Rasch measurement model is applied to the data, only the binary data is used, discarding the information given by the confidence marking. This study shows how Wilson's ordered partition model (OPM), a member of the Rasch family of models, can be used to model the confidence information. The result is a model which is in strict relation to the binary Rasch model, since the Rasch ICC's are "split" into a set of curves each representing a confidence level. The new model provides a set of item parameters that map the probability of being in each confidence level in relation to the test-taker's ability. The study provides a powerful diagnostic tool to assess item difficulty, overconfidence or misuse of confidence levels but also the fact that a question is particularly tricky or creates a lot of doubt.