Automated analysis of common errors in L2 learner production: Prototype web application development

Abstract

This research report presents the development and validation of Auto Error Analyzer, a prototype web application designed to automate the calculation of accuracy and its related metrics for measuring second language (L2) production. Building on recent advancements in natural language processing (NLP) and artificial intelligence (AI), Auto Error Analyzer introduces an automated accuracy measurement component, bridging a gap in existing assessment tools, which traditionally require human judgment for accuracy evaluation. By utilizing a state-of-the-art generative AI model (Llama 3.3) for error detection, Auto Error Analyzer analyzes L2 texts efficiently and cost-effectively, producing accuracy metrics (e.g., errors per 100 words). Validation results demonstrate high agreement between the tool’s error counts and human rater judgments (r = .94), with microaverage precision and recall in error detection being high as well (.96 and .94 respectively, F1 = .95), and its T-unit and clause counts matched outputs from established tools like L2SCA. Developed under open science principles to ensure transparency and replicability, the tool aims to support researchers and educators while emphasizing the complementary role of human expertise in language assessment. The possibilities of Auto Error Analyzer for efficient and scalable error analysis, as well as its limitations in detecting context-dependent and first-language (L1)-influenced errors, are also discussed.