From Item Estimates to Test Operations: The Cascading Effect of Rapid Guessing

Abstract

Inadequate test-taking effort poses a significant challenge, particularly when low-stakes test results inform high-stakes policy and psychometric decisions. We examined how rapid guessing (RG), a common form of low test-taking effort, biases item parameter estimates, particularly the discrimination and difficulty parameters. Previous research reported conflicting findings on the direction of bias and what contributes to it. Using simulated data that replicate real-world, low-stakes testing conditions, this study reconciles the inconsistencies by identifying the conditions under which item parameters are over- or underestimated. Bias is influenced by item-related factors (true parameter values and the number of RG responses the items receive) and examinee-related factors (proficiency differences between rapid guessers and non-rapid guessers, the variability in RG behavior among rapid guessers, and the pattern of RG responses throughout the test). The findings highlight that ignoring RG not only distorts proficiency estimates but may also impact broader test operations, including adaptive testing, equating, and standard setting. By demonstrating the potential far-reaching effects of RG, we underline the need for testing professionals to implement methods that mitigate RG's impact (such as motivation filtering) to protect the integrity of their psychometric work.