Can algorithm-based feedback help students to write better? A meta-analysis exploring surface- and deep-level outcomes

IF 7.6 1区文学 Q1 EDUCATION & EDUCATIONAL RESEARCH

Assessing Writing Pub Date : 2026-04-01 Epub Date: 2026-04-12 DOI:10.1016/j.asw.2026.101034

Sina Scherer , Steve Graham , Vera Busse

{"title":"Can algorithm-based feedback help students to write better? A meta-analysis exploring surface- and deep-level outcomes","authors":"Sina Scherer , Steve Graham , Vera Busse","doi":"10.1016/j.asw.2026.101034","DOIUrl":null,"url":null,"abstract":"<div><div>Against the backdrop of rapid developments of algorithm-based feedback tools — from older tools mainly providing feedback on grammar and spelling to advanced tools based on generative artificial intelligence offering more comprehensive writing support — our meta-analysis examines to what extent algorithm-based feedback improves not only surface- (e.g., grammar and spelling) but also deep-level (e.g., structure, content, coherence) writing outcomes for different learners at secondary school and university. We reviewed experimental and quasi-experimental studies published between 2011 and the end of 2024, covering five European languages. Results from the 33 included studies indicated that algorithm-based feedback was beneficial for improving writing in general (<em>g</em> = 0.36). Specifically, positive effects were observed for surface-level outcomes at posttest (<em>g</em> = 0.31), though no lasting effects were found at maintenance (<em>g</em> = −0.02). In contrast, deep-level writing outcomes showed sustained improvement, with positive effects both at posttest (<em>g</em> = 0.31) and maintenance (<em>g</em> = 0.54). No significant differences between secondary and university students were observed. However, L2 learners, in general, seemed to profit most from algorithm-based feedback, showing gains in surface- (<em>g</em> = 0.77, bordering on significance), and deep-level outcomes (<em>g</em> = 0.46). While no significant differences were found between the effects of specific types of algorithm-based feedback tools, feedback from Grammarly and Pigai statistically enhanced students’ writing, but effects of ChatGPT feedback were non-significant. We discuss implications for future research and educational practice, also in light of the small transfer of learning to new writing tasks.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"68 ","pages":"Article 101034"},"PeriodicalIF":7.6000,"publicationDate":"2026-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Assessing Writing","FirstCategoryId":"98","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S107529352600022X","RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2026/4/12 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 0

Abstract

Against the backdrop of rapid developments of algorithm-based feedback tools — from older tools mainly providing feedback on grammar and spelling to advanced tools based on generative artificial intelligence offering more comprehensive writing support — our meta-analysis examines to what extent algorithm-based feedback improves not only surface- (e.g., grammar and spelling) but also deep-level (e.g., structure, content, coherence) writing outcomes for different learners at secondary school and university. We reviewed experimental and quasi-experimental studies published between 2011 and the end of 2024, covering five European languages. Results from the 33 included studies indicated that algorithm-based feedback was beneficial for improving writing in general (g = 0.36). Specifically, positive effects were observed for surface-level outcomes at posttest (g = 0.31), though no lasting effects were found at maintenance (g = −0.02). In contrast, deep-level writing outcomes showed sustained improvement, with positive effects both at posttest (g = 0.31) and maintenance (g = 0.54). No significant differences between secondary and university students were observed. However, L2 learners, in general, seemed to profit most from algorithm-based feedback, showing gains in surface- (g = 0.77, bordering on significance), and deep-level outcomes (g = 0.46). While no significant differences were found between the effects of specific types of algorithm-based feedback tools, feedback from Grammarly and Pigai statistically enhanced students’ writing, but effects of ChatGPT feedback were non-significant. We discuss implications for future research and educational practice, also in light of the small transfer of learning to new writing tasks.

查看原文本刊更多论文

基于算法的反馈能帮助学生更好地写作吗？探索表层和深层结果的荟萃分析

在基于算法的反馈工具快速发展的背景下，从主要提供语法和拼写反馈的旧工具到基于生成人工智能的先进工具，提供更全面的写作支持我们的荟萃分析考察了基于算法的反馈在多大程度上不仅提高了表层（如语法和拼写），而且提高了深层次（如结构、内容、连贯性）的写作效果，适用于中学和大学的不同学习者。我们回顾了2011年至2024年底期间发表的实验和准实验研究，涵盖了五种欧洲语言。纳入的33项研究的结果表明，基于算法的反馈总体上有利于提高写作水平（g = 0.36）。具体来说，在测试后，表面水平的结果观察到积极的影响（g = 0.31），尽管在维持时没有发现持久的影响（g = - 0.02）。相比之下，深度写作结果显示持续改善，在后测（g = 0.31）和维持（g = 0.54）均有积极影响。中学生与大学生之间无显著差异。然而，总的来说，二语学习者似乎从基于算法的反馈中获益最多，在表层（g = 0.77，接近显著性）和深层结果（g = 0.46）上都有收获。虽然特定类型的基于算法的反馈工具之间的效果没有显著差异，但Grammarly和Pigai的反馈在统计上提高了学生的写作水平，而ChatGPT的反馈效果不显著。我们讨论了对未来研究和教育实践的影响，也考虑到学习到新的写作任务的小转移。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Assessing Writing Multiple-

CiteScore

6.00

自引率

17.90%

发文量

期刊介绍： Assessing Writing is a refereed international journal providing a forum for ideas, research and practice on the assessment of written language. Assessing Writing publishes articles, book reviews, conference reports, and academic exchanges concerning writing assessments of all kinds, including traditional (direct and standardised forms of) testing of writing, alternative performance assessments (such as portfolios), workplace sampling and classroom assessment. The journal focuses on all stages of the writing assessment process, including needs evaluation, assessment creation, implementation, and validation, and test development.