A comparative analysis of AI-generated texts, corpus data, and speaker judgments: Subject honorification patterns in Korean

Abstract

Technological innovations can greatly enhance second language (L2) pragmatics instruction by providing learners with more natural and authentic communication opportunities. As Generative Artificial Intelligence (GenAI) tools become increasingly integrated into L2 teaching, questions arise as to whether they provide pedagogically appropriate input and how they can be used for inductive instruction (e.g., Data-driven Learning). To advance meaningful instructional approaches to Korean honorifics, understanding the nature of input is key; particularly, what exemplars of honorifics are available through GenAI and spoken corpora and how L2 learners perceive and evaluate different honorific forms. In response to these inquiries, we analyzed patterns of subject-verb honorific agreement in outputs from ChatGPT 4.0 and the NIKL Korean Dialogue Summarization Corpus (Study 1), and conducted an acceptability judgment test of four subject-verb honorific (mis)match forms (Study 2). We found that ChatGPT predominantly favored a subject-verb matched form, whereas corpus data reflected the highly complex, context-dependent use and variations of honorifics. L1 judgments aligned more closely with the corpus results, reflecting sensitivity to nuanced (mis)match forms, whereas L2 judgments closely mirrored ChatGPT’s patterns, lacking sensitivity beyond the matched forms. These results underscore the challenges associated with Korean honorification for both learners and educators, highlighting the need for more refined inductive teaching.