An Efficient Non-parametric Item Selection Method for Polytomous Scoring CD-CAT

Abstract

In educational evaluations at home and abroad, polytomous scoring items are becoming increasingly important. They can provide richer and more valuable information, with unmatched advantages compared to binary (0-1 scoring) items. If used as a tool for teachers to diagnose and assess students in the classroom, CD-CAT (Cognitive Diagnostic Computerized Adaptive Testing) has significant implications for improving teaching effectiveness. However, in classroom teaching situations, it is not feasible to estimate item parameters accurately with a large sample, as in large-scale assessments. In such cases, non-parametric CD-CAT becomes the only viable option. Compared to parametric CD-CAT, non-parametric CD-CAT started later and is particularly lacking in research related to polytomous scoring. Item selection method is at the core of CD-CAT, so it is essential to develop a non-parametric item selection method suitable for polytomous scoring CD-CAT. This study proposes a non-parametric item selection method for polytomous scoring cognitive diagnostic computerized adaptive testing (PCD-CAT): the Manhattan Distance Non-parametric Difference index item selection method (MD-NDI). The results of simulation studies indicate: (1) MD-NDI item selection method is suitable for PCD-CAT scenarios and exhibits better performance when the item bank quality is poor or the sample size for estimating item parameters is limited. (2) MD-NDI does not require pre-testing of items and distributes item usage more evenly, effectively ensuring the security of the item bank. (3) Even in cases of incorrectly specified item bank of Qc-matrix, MD-NDI still shows higher pattern correct classification rates. (4) In the study of variable-length PCD-CAT, MD-NDI not only reduces the test length in most conditions but also has a higher pattern correct classification rates when reaching the test termination rule.