A new Q-matrix validation method based on signal detection theory.

Abstract

The Q-matrix is a crucial component of cognitive diagnostic theory and an important basis for the research and practical application of cognitive diagnosis. In practice, the Q-matrix is typically developed by domain experts and may contain some misspecifications, so it needs to be refined using Q-matrix validation methods. Based on signal detection theory, this paper puts forward a new Q-matrix validation method (i.e., $\beta$ method) and then conducts a simulation study to compare the new method with existing methods. The results show that when the model is DINA (deterministic inputs, noisy 'and' gate), the $\beta$ method outperforms the existing methods under all conditions; under the generalized DINA (G-DINA) model, the method still has the highest validation rate when the sample size is small, and the item quality is high or the rate of Q-matrix misspecification is ≥.4. Finally, a sub-dataset of the PISA 2000 reading assessment is analysed to evaluate the reliability of the $\beta$ method.