Learning and Engaging With AI: An Exploration of the Effects of Mental Workload and Search Behavior on Short-Term Computer-Based Learning With a Chatbot.

Abstract

ObjectiveThis study examined how learning, workload and search behaviors were impacted by a chatbot during a self-regulated Web search task, as opposed to more classic search engines.BackgroundArtificial intelligence technologies, including chatbots, are becoming increasingly accessible. These tools have been demonstrated useful to support self-regulated learning, mostly in structured learning contexts. The reduction in workload they offer may, however, prevent key learning strategies from being deployed, especially for Web search.MethodSixty participants were asked to answer a set of essay questions and to rate their workload, effort deployed and literacy while either gathering information from the Internet (Web condition) or by chatting with a chatbot (LLM condition) with the possibility of verifying information on the Web. Several key search strategies and chatbot interaction measures were extracted. A surprise memory test was also presented to evaluate how they learned the content addressed in the essay questions.ResultsMeasures of effort, mental workload, search behaviors and literacy differed significantly across conditions. Performance on the memory test did not vary. Multiple relationships with memory performance, including key Web search and verification behaviors, were found.ConclusionChatbots may help reduce workload and short-term learning with a chatbot may be more influenced by the nature of the interaction with the tool, rather than the tool itself.ApplicationEffective uses of chatbots may require learners to verify the content generated by the chatbot and to show superior engagement. Engagement-promoting learning activities should be considered when using LLM-driven agents to support self-regulated, Web-based search.