Chatbots in education: Hype or help? A meta-analysis

IF 9 1区心理学 Q1 PSYCHOLOGY, EDUCATIONAL

Learning and Individual Differences Pub Date : 2025-02-11 DOI:10.1016/j.lindif.2025.102646

Martin Laun, Fabian Wolff

{"title":"Chatbots in education: Hype or help? A meta-analysis","authors":"Martin Laun, Fabian Wolff","doi":"10.1016/j.lindif.2025.102646","DOIUrl":null,"url":null,"abstract":"<div><div>Artificial intelligence integration in education, primarily through chatbots, has emerged as a potential solution to address the challenges of catering to students' diverse learning backgrounds. This meta-analysis examined chatbot effectiveness in education, driven by amplified interest since ChatGPT's introduction in 2022. Adhering to PRISMA guidelines, we analyzed 135 effect sizes from 62 studies across diverse educational contexts. Initial results revealed a large positive effect of chatbots on learning performance. However, after controlling for publication bias, we found a significant small to moderate effect. Interaction mode, field of study, intervention duration, and educational level emerged as significant moderators. Text-based interactions, STEM disciplines, longer interventions, and lower educational levels showed the most substantial chatbot effectiveness in enhancing learning performance. Neither chatbot type nor cultural differences significantly influenced learning performance. While our findings support chatbots' potential to enhance performance, they underscore the necessity for targeted implementation when integrating these technologies into learning environments.</div></div><div><h3>Educational relevance statement</h3><div>Our meta-analysis reveals that chatbots have a small to moderate statistically significant positive effect on student learning performance. While chatbots show promise in enhancing education, particularly in STEM fields, text-based interactions, and for longer-term interventions, their impact may be overestimated due to publication bias. These findings underscore the need for targeted chatbot integration in education. Further research is essential as AI continues to evolve to maximize chatbots' educational benefits.</div></div>","PeriodicalId":48336,"journal":{"name":"Learning and Individual Differences","volume":"119 ","pages":"Article 102646"},"PeriodicalIF":9.0000,"publicationDate":"2025-02-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Learning and Individual Differences","FirstCategoryId":"102","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1041608025000226","RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"PSYCHOLOGY, EDUCATIONAL","Score":null,"Total":0}

引用次数: 0

Abstract

Artificial intelligence integration in education, primarily through chatbots, has emerged as a potential solution to address the challenges of catering to students' diverse learning backgrounds. This meta-analysis examined chatbot effectiveness in education, driven by amplified interest since ChatGPT's introduction in 2022. Adhering to PRISMA guidelines, we analyzed 135 effect sizes from 62 studies across diverse educational contexts. Initial results revealed a large positive effect of chatbots on learning performance. However, after controlling for publication bias, we found a significant small to moderate effect. Interaction mode, field of study, intervention duration, and educational level emerged as significant moderators. Text-based interactions, STEM disciplines, longer interventions, and lower educational levels showed the most substantial chatbot effectiveness in enhancing learning performance. Neither chatbot type nor cultural differences significantly influenced learning performance. While our findings support chatbots' potential to enhance performance, they underscore the necessity for targeted implementation when integrating these technologies into learning environments.

Educational relevance statement

Our meta-analysis reveals that chatbots have a small to moderate statistically significant positive effect on student learning performance. While chatbots show promise in enhancing education, particularly in STEM fields, text-based interactions, and for longer-term interventions, their impact may be overestimated due to publication bias. These findings underscore the need for targeted chatbot integration in education. Further research is essential as AI continues to evolve to maximize chatbots' educational benefits.

查看原文本刊更多论文

教育领域的聊天机器人：炒作还是帮助？一个荟萃分析

人工智能在教育中的整合，主要是通过聊天机器人，已经成为解决迎合学生不同学习背景挑战的潜在解决方案。这项荟萃分析考察了聊天机器人在教育中的有效性，这是由于自2022年ChatGPT推出以来人们对聊天机器人的兴趣不断扩大。遵循PRISMA指南，我们分析了来自62项不同教育背景的研究的135个效应量。初步结果显示，聊天机器人对学习成绩有很大的积极影响。然而，在控制了发表偏倚后，我们发现了显著的小到中等的影响。互动模式、研究领域、干预持续时间和教育水平是显著的调节因素。基于文本的交互、STEM学科、较长的干预和较低的教育水平显示了聊天机器人在提高学习成绩方面最显著的有效性。聊天机器人类型和文化差异对学习表现都没有显著影响。虽然我们的研究结果支持聊天机器人提高性能的潜力，但它们强调了在将这些技术集成到学习环境中时有针对性实施的必要性。教育相关性声明我们的荟萃分析显示，聊天机器人对学生的学习表现有小到中等的统计学显著的积极影响。虽然聊天机器人在加强教育，特别是在STEM领域、基于文本的互动以及长期干预方面表现出了希望，但由于发表偏见，它们的影响可能被高估了。这些发现强调了在教育中有针对性地整合聊天机器人的必要性。随着人工智能的不断发展，为了最大限度地提高聊天机器人的教育效益，进一步的研究是必不可少的。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Learning and Individual Differences PSYCHOLOGY, EDUCATIONAL-

CiteScore

6.60

自引率

2.80%

发文量

期刊介绍： Learning and Individual Differences is a research journal devoted to publishing articles of individual differences as they relate to learning within an educational context. The Journal focuses on original empirical studies of high theoretical and methodological rigor that that make a substantial scientific contribution. Learning and Individual Differences publishes original research. Manuscripts should be no longer than 7500 words of primary text (not including tables, figures, references).