Providing tailored reflection instructions in collaborative learning using large language models

Abstract

The relative effectiveness of reflection either through student generation of contrasting cases or through provided contrasting cases is not well-established for adult learners. This paper presents a classroom study to investigate this comparison in a college level Computer Science (CS) course where groups of students worked collaboratively to design database access strategies. Forty-four teams were randomly assigned to three reflection conditions ([GEN] directive to generate a contrasting case to the student solution and evaluate their trade-offs in light of the principle, [CONT] directive to compare the student solution with a provided contrasting case and evaluate their trade-offs in light of a principle, and [NSI] a control condition with a non-specific directive for reflection evaluating the student solution in light of a principle). In the CONT condition, as an illustration of the use of LLMs to exemplify knowledge transformation beyond knowledge construction in the generation of an automated contribution to a collaborative learning discussion, an LLM generated a contrasting case to a group's solution to exemplify application of an alternative problem solving strategy in a way that highlighted the contrast by keeping many concrete details the same as those the group had most recently collaboratively constructed. While there was no main effect of condition on learning based on a content test, low-pretest student learned more from CONT than GEN, with NSI not distinguishable from the other two, while high-pretest students learned marginally more from the GEN condition than the CONT condition, with NSI not distinguishable from the other two.

Practitioner notes

What is already known about this topic

• Reflection during or even in place of computer programming is beneficial for learning of principles for advanced computer science when the principles are new to students.
• Generation of contrasting cases and comparing contrasting cases have both been demonstrated to be effective as opportunities to learn from reflection in some contexts, though questions remain about ideal applicability conditions for adult learners.
• Intelligent conversational agents can be used effectively to deliver stimuli for reflection during collaborative learning, though room for improvement remains, which provides an opportunity to demonstrate the potential positive contribution of large language models (LLMs).

What this paper adds

• The study contributes new knowledge related to the differences in applicability conditions between generation of contrasting cases and comparison across provided contrasting cases for adult learning.
• The paper presents an application of LLMs as a tool to provide contrasting cases tailored to the details of actual student solutions.
• The study provides evidence from a classroom intervention study for positive impact on student learning of an LLM-enabled intervention.

Implications for practice and/or policy

• Advanced computer science curricula should make substantial room for reflection alongside problem solving.
• Instructors should provide reflection opportunities for students tailored to their level of prior knowledge.
• Instructors would benefit from training to use LLMs as tools for providing effective contrasting cases, especially for low-prior-knowledge students.