A new person-fit statistic for the detection of aberrant responses in polytomous cognitive diagnostic models.

Abstract

Assessing person-fit in cognitive diagnostic assessments is a critical research area. Inability to identify misfitting responses can lead to misinterpretation of students' attribute profiles, potentially resulting in incorrect remedial actions. Despite its importance, there is a lack of research on person-fit statistics for polytomous cognitive diagnostic models (CDM). To address this, we propose a new person-fit statistic, WR, specifically designed for polytomous items in CDMs. We evaluated WR's ability to detect three types of abnormal behaviors through simulation studies, comparing its performance with established statistics including l_z, infit, and outfit. The results show that WR consistently demonstrated stable and superior detection capabilities across all experimental scenarios. Traditional methods showed inconsistent detection abilities for different anomalies; l_z was more effective at detecting cheating, while infit was better for creative responses. In high-quality test environments, WR performed best, though the difference compared to traditional methods was not significant. However, in low-quality conditions, WR significantly outperformed traditional methods. Overall, WR proved to be an effective tool for detecting person misfit in polytomous scoring CDMs. Finally, we analyzed a real educational assessment data to assess the practical application of WR.