Cognitive Echo: Enhancing think-aloud protocols with LLM-based simulated students

Abstract

In the field of education, the think-aloud protocol is commonly used to encourage learners to articulate their thoughts during the learning process, providing observers with valuable insights into learners' cognitive processes beyond the final learning outcomes. However, the implementation of think-aloud protocols faces challenges such as task interference and limitations in completeness and authenticity of verbal reports. This study proposes a method called Cognitive Echo, which leverages large language models (LLMs) trained with simulated student experiences to enhance the completeness and authenticity of think-aloud verbalizations. LLMs have been demonstrated to simulate human-like behaviour more effectively by memorizing experiences. In this work, we introduce specific learner roles and train the LLMs to act as distinct learners. Our method involves integrating transaction data from learners' interactions with a tutoring system and the tutor's content to create interactive experiences between learners and teachers, thereby training the model to become simulated students with learning experiences. To investigate the effectiveness of this approach, we designed a test playground based on the retrospective think-aloud protocol and examined how LLM-trained simulated students improve cognitive process transparency and generalization of learning strategies. The study found that Cognitive Echo not only reveals what simulated students genuinely think about their learning experiences but also enables them to transfer their different cognitive strategies to new tasks. By training simulated students on real learning behaviour data to ensure their cognitive processes reflect authentic learner experiences, this approach will extend think-aloud protocols to more practice-oriented applications.

Practitioner notes

What is already known about this topic

• Think-aloud protocols are widely used in educational settings to explore students' cognitive processes by asking them to verbalize their thoughts while solving problems, but they are prone to issues like task interference and incomplete data reporting.
• Existed applications of simulating student cognition in educational research are rigid and less adaptive to individual learner characteristics.
• Artificial intelligences, especially large language models, have shown promise in educational contexts, particularly for simulating human-like behaviours.

What this paper adds

• This paper introduces the concept of Cognitive Echo, a method that integrates LLM-powered simulated students into think-aloud protocols, which addresses the limitations of traditional verbalization-based methods by leveraging retrospective data.
• The study shows that LLMs, when fine-tuned with authentic learner experiences, can replicate distinct human-like cognitive processes, enabling a more complete and authentic simulation of how students think and solve problems.
• It demonstrates how the use of LLMs to simulate students' cognitive processes can enhance the transparency and completeness of think-aloud protocols by allowing researchers to capture cognitive strategies and behaviours that would otherwise go unspoken.

Implications for practice and/or policy

• Teacher training programmes can benefit from integrating LLM-based simulated students, which enable preservice teachers to practice responding to a wide range of cognitive processes and challenges without the constraints of real-time think-aloud tasks.
• The Cognitive Echo method, by offering a more authentic and less intrusive way of capturing student cognition, can be applied in teacher training scenarios where simulation of real-world classroom dynamics is crucial for developing pedagogical skills.
• The use of Cognitive Echo could help in the creation of digital twins of educational scenarios, facilitating research into complex educational issues (eg, bullying and learning disabilities) through simulations that model real-world interactions.