Evaluating GPT ratings of EFL writing: A scoping review

Abstract

The rise of large language models (LLMs), exemplified by GPT, has opened new possibilities for automated essay scoring (AES) in L2 education. Over the past three years, a growing number of studies have investigated GPT’s potential as a rater of English as a foreign language (EFL) writing. However, with no synthesis existing, the literature remains relatively fragmented. To address this gap, this scoping review analyzed 26 identified studies in terms of their research designs, evaluation foci, reported findings, and summative evaluations. Collectively, these studies addressed three core aspects of the evaluation inference in a full validity argument—accuracy, consistency, and fairness, and presented a cautiously positive view of GPT’s performance in rating EFL essays. Preliminary insights include GPT-4 and GPT-4o’s superiority over standard GPT-3.5 in accuracy and consistency, the promise of few-shot learning prompts, and GPT’s tendency to score more severely in language-related dimensions. The review also identified some methodological limitations across the literature, and highlighted key areas for further investigation. By providing a structured overview of this emerging field for the first time, this scoping review offers guidance for future research and for L2 educators considering the use of GPT models in EFL writing assessment.