Automated Q-matrix Identification Using Text Classification Techniques

Cognitive diagnosis can be very useful to teachers and students, yet its application is limited so far because the current method to identify the Q-matrix is labor intensive and time-consuming. In this study, we propose to use text classification techniques to automatically identify Q-matrix. Specifically, we developed a three-stage model. In the first stage, a compact set of key features which are helpful in identifying different cognitive attributes are selected from items. In the second stage, items are transformed into real-valued vectors based on the key features for machine learning. In the third stage, three machine learning algorithms, logistic regression, support vector machine and Naive Bayes, are used and compared for automated Q-matrix identification. Using a sample of 805 third grade math items, we found Naive Bayes was the best algorithm, yielding an accuracy of 85.2% and an F1 measure of 85.6%. Our result indicated that text classification methods have great potential to automatically identify Q-matrix efficiently, and in turn, make cognitive diagnostic more feasible to practitioners.