Integrating move analysis and sentence reconstruction in automated writing evaluation for L2 academic writers

Abstract

Artificial intelligence has been widely utilized to assist L2 writers through automated writing evaluation (AWE) systems, which offer grammatical feedback. However, for English academic writing, such feedback is apparently insufficient to address the complexities of academic discourse. While genre-based AWE systems employ move analysis, they offer move detections as corrective feedback (CF) without addressing language use issues and are developed using limited datasets. Additionally, general-purpose large language models (LLMs; e.g., ChatGPT) may lack specialized mechanisms for accurately identifying rhetorical moves and providing genre-specific feedback in academic writing contexts. To address these limitations, this study proposes GURUS, a genre-based AWE system grounded in second language acquisition theories. It provides indirect CF by classifying moves with probabilistic scores, and direct CF through sentence reconstruction. GURUS is implemented as a web-based application using ensemble learning model and transformer-based LLMs. By offering indirect and direct CF, GURUS promotes learner-machine interaction, prompting learners to notice discrepancies between their writing and the reconstructed sentences. GURUS was trained on over one million sentences with OMRC moves. Its classification performance was assessed using F1-score and Brier score; furthermore, semantic and rhetorical production were evaluated using BERTscore and human assessment. The results show that GURUS sufficiently classifies sentence moves and reconstructs sentences while retaining semantic integrity. Given GURUS holds promise in academic writing instruction, this study also discusses its implementation to bolster learners’ genre awareness and proficiency in move-based abstract writing.