Detecting Careless Cases in Practice Tests

Abstract

In this paper, we present a novel method for detecting careless responses in a low-stakes practice exam using machine learning models. Rather than classifying test-taker responses as careless based on model fit statistics or knowledge of truth, we built a model to predict significant changes in test scores between a practice test and an official test based on attributes of practice test items. We extracted features from practice test items using hypotheses about how careless test takers respond to items and cross-validated model performance to optimize out-of-sample predictions and reduce heteroscedasticity when predicting the closest official test. All analyses use data from the practice and official versions of the Duolingo English Test. We discuss the implications of using a machine learning model for predicting careless cases as compared with alternative, popular methods.