Assigning CEFR-J levels to English learners’ writing: An approach using lexical metrics and generative AI

This study presents the CEFR-based Writing Level Analyzer (CWLA), a novel automated system for assessing English learners’ writing proficiency. It assesses proficiency according to CEFR-J levels, a finely graded adaptation of the CEFR framework tailored to the English as a Foreign Language context, particularly in Japan. CWLA generates scores by combining vocabulary scores with AI-based analytical scores, allowing for a sophisticated, regression-based alignment with CEFR-J proficiency levels. By leveraging both straightforward lexical metrics and advanced AI scoring, CWLA provides accurate and interpretable assessments accessible via a user-friendly web interface. To evaluate CWLA's effectiveness, we conducted validation using the ICNALE GRA dataset, which showed a strong correlation of 0.88 between CWLA scores and human ratings. Additionally, entropy analysis indicated that CWLA's scoring distribution closely resembles human rater patterns, capturing the variability expected in human assessments. Three CEFR/CEFR-J specialists also performed expert validation, resulting in an agreement rate of 83.33 %, thereby supporting the system's alignment with expert judgment. These results suggest that CWLA is a reliable system for detailed CEFR-J level assessment, offering promising applications in language education and learner assessment across proficiency levels.