Using text mining and machine learning to predict reasoning activities from think-aloud transcripts in computer assisted learning

Coding think-aloud transcripts is time-consuming and labor-intensive. In this study, we examined the feasibility of predicting students’ reasoning activities based on their think-aloud transcripts by leveraging the affordances of text mining and machine learning techniques. We collected the think-aloud data of 34 medical students as they diagnosed virtual patients in an intelligent tutoring system. The think-aloud data were transcribed and segmented into 2,792 meaningful units. We used a text mining tool to analyze the linguistic features of think-aloud segments. Meanwhile, we manually coded the think-aloud segments using a medical reasoning coding scheme. We then trained eight types of supervised machine learning algorithms to predict reasoning activities based on the linguistic features of students’ think-aloud transcripts. We further investigated if the performance of prediction models differed between high and low performers. The results suggested that students’ reasoning activities could be predicted relatively accurately by the linguistic features of their think-aloud transcripts. Moreover, training the predictive models using the data instances of either high or low performers did not lower the models’ performance. This study has significant methodological and practical implications regarding the automatic analysis of think-aloud protocols and real-time assessment of students’ reasoning activities.