Investigating the capabilities of large language model-based task-oriented dialogue chatbots from a learner’s perspective

Abstract

Second language (L2) learning has explored ways to maximize the benefits of using chatbots as a pedagogical resource for language practice and development. Given the growing consensus regarding the need to develop task-oriented dialogue (TOD) chatbots specifically designed for language learning purposes and recent advances in large language models (LLMs), the present study investigates the capabilities of LLM-based TOD chatbots from L2 learners’ perspectives. To this end, we developed two TOD conversational agents using Poe Artificial Intelligence (AI), which provides easy-to-use LLM-based chatbot-building tools. South Korean undergraduate L2 students were asked to engage in two chatbot-based linguistic tasks and complete a survey regarding their perceptions of LLM-based TOD chatbots. The results showed that the utterances generated by these state-of-the-art chatbots were perceived as very natural, and they were found to be highly capable of understanding dialogues in context and keeping track of the progress of the conversation to produce contextually appropriate responses to the user’s input. These chatbots were rated positively overall in terms of their usefulness as a resource for L2 learning, as learners are motivated and remain interested in engaging in conversations with chatbots, thus maximizing learning effects.