Automated vs. manual linguistic annotation for assessing pragmatic competence in English classes

Evaluating pragmatic competence remains a complex and critical challenge in applied linguistics, particularly in English as a Foreign Language (EFL) contexts. This study aims to address this gap by examining the potential of automating pragmatic competence assessment using AI-powered text analytics. Employing an explanatory sequential mixed-methods design, the quantitative phase compares the accuracy of automated versus manual linguistic annotation in evaluating the pragmatic skills of EFL learners. In the qualitative phase, factors influencing the accuracy of manual annotation are explored. For automated annotation, ChatGPT-4 Omni (ChatGPT-4o) processed 116 transcriptions representing participants' performances across six verbal discourse completion tasks (DCTs), encompassing prosodic features and pragmatic functions such as requesting favors, apologizing, suggesting, complaining, inviting, and refusing invitations. The AI model was fine-tuned using a human-in-the-loop approach, incorporating ensemble techniques such as few-shot learning and instructional prompts. Manual annotation employed trained EFL teachers using standardized assessment cards. Results indicate that automated annotation surpasses manual accuracy in evaluating most pragmatic components, except cultural norms, where both methods exhibit limitations. Focus group findings reveal that annotator bias, fatigue, technological influences, linguistic background differences, and subjectivity impact manual annotation accuracy. This interdisciplinary investigation expands the methodological toolkit for pragmatic competence evaluation and holds significant implications for fields such as digital humanities, computational pragmatics, language education, machine learning, and natural language processing.