The role of critical thinking on undergraduates' reliance behaviours on generative AI in problem-solving

Abstract

There is a heightened concern over undergraduate students being over-reliant on Generative AI and using it recklessly. Reliance behaviours describe the frequencies and ways that people use AI tools for tasks such as problem-solving, influenced by individual factors such as trust and AI literacy. One way to conceptualise reliance is that reliance behaviours are affected by the extent to which learners consciously evaluate the relative performance of AI and humans, suggesting the potential impacts of critical thinking on reliance. This study, thus, empirically investigates the relationship between critical thinking and reliance behaviours. Critical thinking includes disposition and skills. However, limited empirical studies have investigated how critical thinking influences learners' reliance behaviours when solving problems with Generative AI. Hence, the current study conducted path analyses to investigate how critical thinking is associated with reliance behaviours and how it mediates the effect of individual factors on reliance behaviours. We collected 808 survey responses on critical thinking disposition and skills, reliance behaviours (a self-developed and validated scale, including reflective use, cautious use, thoughtless use, and collaborative use), trust towards AI, and AI literacy from undergraduates after a problem-solving task with Generative AI. The results indicate that (1) critical thinking is positively associated with the collaborative, reflective, and cautious use of Generative AI, suggesting that these three types of use of Generative AI could be considered desirable behaviours in human–AI problem-solving; (2) trust positively predicts thoughtless use; (3) critical thinking can offset the influence of trust on collaborative, reflective and cautious use; and (4) critical thinking can amplify the influence of AI literacy on reflective, cautious and collaborative use. This study contributes new insights into understanding the role of critical thinking in fostering desirable reliance behaviours, including reflective, cautious and collaborative use, and provides implications for future interventions when applying Generative AI for problem-solving.

Practitioner notes

What is already known about this topic?

• Generative AI tools can potentially enhance problem-based learning (PBL) by supporting brainstorming and solution refinement.
• Reliance behaviours in human-AI collaboration are influenced by factors such as trust in AI and AI literacy.
• Strategy-graded reliance emphasizes the reasoning process leading to reliance behaviours, focusing on thoughtful engagement with AI tools, and this cognitive process can be captured by critical thinking.

What this paper adds?

• Critical thinking is positively associated with the reflective, collaborative, and cautious use of Generative AI.
• Critical thinking mediates the effects of trust and AI literacy on reliance behaviours, amplifying reflective, cautious and collaborative use while mitigating the thoughtless use of Generative AI.
• The study introduces a nuanced understanding of reliance behaviours by applying a strategy-graded framework, emphasising cognitive engagement rather than a purely outcome-based understanding of reliance behaviours.

Implications for practice and/or policy

• Educational interventions could consider critical thinking when integrating AI tools in problem-solving contexts.
• Students' trust in AI needs to be balanced with critical thinking skills to reduce overreliance and enhance thoughtful engagement with AI tools.