ChatGPT’s Capabilities in Spotting and Analyzing Writing Errors Experienced by EFL Learners

Abstract

The recent Large Language Models (LLMs) use advanced algorithms to identify areas where sentence structure and word choice can be improved and to detect grammar, syntax, and spelling mistakes in sentences. This study aimed to investigate the effectiveness of the Chat Generative Pre-trained Transformer (ChatGPT) in detecting English as a foreign language (EFL) learners’ writing errors compared to human instructors. This study examines the ChatGPT as a recent and advanced LLM in analyzing and processing EFL learners’ writing issues. This paper provides valuable insights into the potential benefits and challenges of integrating Artificial Intelligence (AI) into EFL writing education. Our results revealed that ChatGPT successfully identified most surface-level errors but could not detect writing errors related to deep structure and pragmatics. Conversely, human teachers could spot most of these issues. These findings suggest that while ChatGPT can be a valuable tool in identifying surface-level errors, it cannot replace human instructors’ expertise and nuanced understanding in detecting errors related to the more complex aspects of writing. The writing error types (data) are statistically analyzed. The descriptive analysis displays valuable insights into the reliability of the data and its potential implications, where the F-score, which measures the statistical model accuracy, is found to be 1.5. In the meantime, the p-value score, which shows the probability of obtaining results as extreme as the detected data, is calculated to be 0.23. The results suggest that the collected data is statistically significant, and further analysis may yield valuable insights.