Recovering Accuracy From Miscalibrated Item Banks: Scoring Error Is Fixable, Selection Error Is Not.

Abstract

Computerized adaptive testing (CAT) depends on accurately calibrated item banks, but in practice, the item parameters governing deployment of the item banks are often based on small pilot samples, transported across populations, or assigned placeholder values pending proper calibration. Throughout, miscalibration is modeled in the item difficulty parameters, with discriminations held at their true values. We decompose the impact of miscalibration on subsequent ability estimates into two components: item selection error and scoring error. While selection error is not recoverable, rescoring using better calibrations, and without any additional testing burden, can potentially provide a post hoc fix to scoring errors. In a simulation study, we find that rescoring substantially reduces root mean squared error, with scoring error accounting for 86% to 96% of miscalibration-attributable error. In Study 2, we apply this framework to data from 2,657 children (ages 5-12) assessed across three pilot sites in the LEVANTE project. Using identical scoring methods with old versus new item parameters, we found that rescoring shifted scores substantially and reordered children's relative standing; external validation confirms the rescored estimates are better, not just different. Low-ability and younger children are disproportionately affected by rescoring, underscoring the equity implications. These results highlight the utility of rescoring for any CAT using early-stage, transported, or AI-generated item parameters, and how rescoring can enable rapid deployment of assessments to new populations.