The effect of chatbots on learning: a meta-analysis of empirical research

Abstract

This meta-analysis aimed to comprehensively review empirical studies on the effect of chatbots on learning and quantitatively synthesize their findings to produce an overall effect size. Searching several databases yielded 28 eligible reports with 31 individual effect sizes. The results revealed a significant and medium effect (g = .48) of chatbots on learning. Further analyses indicated four significant moderators: the type of instruction in the comparison group, experimental duration, chatbot type, and chatbot tasks. The highest effect sizes emerged when the comparison group had no specific support, the experiment lasted only one session, and chatbots were task-focused and took care of frequently asked questions. These results suggest that chatbots can be more effective in certain cases within their overall contribution area to learning.