The effects of critical thinking intervention on reliance behaviors, problem-solving quality, and creativity during human-Generative AI collaborative learning

Abstract

As Generative AI becomes increasingly used in various educational contexts, understanding how students engage with these tools during collaborative problem-solving is critical. While prior research suggests that critical thinking is essential in human-AI problem-solving, few studies have examined how instructional interventions, targeting critical thinking, might shape their reliance behaviors and collaborative outcomes. This study investigates the effects of a critical thinking intervention embedded in a problem-based learning (PBL) environment where students are engaged with Generative AI. The intervention combined strategies that foster critical thinking, including authentic instruction, structured dialogue, and AI-supported peer mentoring, aiming to promote students' thoughtful engagement and improve problem-solving performance. Participants (N = 226) were assigned to experimental (with critical thinking interventions) or comparison (without critical thinking interventions) conditions. We used pre- and post-surveys to measure participants' trust, critical thinking, and AI reliance behaviors, and group reports and chat histories to assess their problem-solving quality and creativity. Results revealed that the intervention did not produce significant improvement in self-reported critical thinking, possibly due to the short intervention duration. However, the intervention led to a marginal reduction in students' thoughtless use of Generative AI and significantly reduced the direct adoption of AI-generated content. Notably, students in the intervention condition produced more creative solutions, demonstrating higher levels of originality and idea density in their group reports. These findings suggest that how students use Generative AI is critical, especially when it is almost impossible to control whether they use it or not. The study highlights the importance of designing interventions that cultivate students’ critical thinking to support creative human-AI problem-solving.