The Effectiveness of Automated Writing Evaluation on Writing Quality: A Meta-Analysis

IF 4 2区教育学 Q1 EDUCATION & EDUCATIONAL RESEARCH

Journal of Educational Computing Research Pub Date : 2022-11-17 DOI:10.1177/07356331221127300

Nasha Zhai, Xiaomei Ma

{"title":"The Effectiveness of Automated Writing Evaluation on Writing Quality: A Meta-Analysis","authors":"Nasha Zhai, Xiaomei Ma","doi":"10.1177/07356331221127300","DOIUrl":null,"url":null,"abstract":"Automated writing evaluation (AWE) has been frequently used to provide feedback on student writing. Many empirical studies have examined the effectiveness of AWE on writing quality, but the results were inconclusive. Thus, the magnitude of AWE’s overall effect and factors influencing its effectiveness across studies remained unclear. This study re-examined the issue by meta-analyzing the results of 26 primary studies with a total of 2468 participants from 2010 to 2022. The results revealed that AWE had a large positive overall effect on writing quality (g = 0.861, p < 0.001). Further moderator analyses indicated that AWE was more effective for post-secondary students than for secondary students and had more benefits for English as a Foreign Language (EFL) and English as a Second Language (ESL) learners than for Native English Speaker (NES) learners. When the genre of writing was considered, AWE showed a more significant impact on argumentative writing than on academic and mixed writing genres. However, intervention duration, feedback combination, and AWE platform did not moderate the effect of AWE on writing quality. The implications and recommendations for both research and practice are discussed in depth.","PeriodicalId":47865,"journal":{"name":"Journal of Educational Computing Research","volume":"61 1","pages":"875 - 900"},"PeriodicalIF":4.0000,"publicationDate":"2022-11-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"9","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Educational Computing Research","FirstCategoryId":"95","ListUrlMain":"https://doi.org/10.1177/07356331221127300","RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 9

Abstract

Automated writing evaluation (AWE) has been frequently used to provide feedback on student writing. Many empirical studies have examined the effectiveness of AWE on writing quality, but the results were inconclusive. Thus, the magnitude of AWE’s overall effect and factors influencing its effectiveness across studies remained unclear. This study re-examined the issue by meta-analyzing the results of 26 primary studies with a total of 2468 participants from 2010 to 2022. The results revealed that AWE had a large positive overall effect on writing quality (g = 0.861, p < 0.001). Further moderator analyses indicated that AWE was more effective for post-secondary students than for secondary students and had more benefits for English as a Foreign Language (EFL) and English as a Second Language (ESL) learners than for Native English Speaker (NES) learners. When the genre of writing was considered, AWE showed a more significant impact on argumentative writing than on academic and mixed writing genres. However, intervention duration, feedback combination, and AWE platform did not moderate the effect of AWE on writing quality. The implications and recommendations for both research and practice are discussed in depth.

查看原文本刊更多论文

自动写作评估对写作质量的有效性:一项元分析

自动写作评估（AWE）经常被用来提供对学生写作的反馈。许多实证研究都检验了AWE对写作质量的有效性，但结果并不确定。因此，AWE的总体效应的大小以及影响其在各研究中有效性的因素尚不清楚。这项研究通过对2010年至2022年共有2468名参与者的26项初级研究的结果进行荟萃分析，重新审视了这一问题。研究结果表明，AWE对写作质量有很大的总体正向影响（g=0.861，p<0.001）。进一步的调节因素分析表明，与母语为英语的学习者相比，AWE对于中学后学生更有效，并且对英语作为外语（EFL）和英语作为第二语言（ESL）的学习者更有好处。当考虑写作类型时，AWE对议论文写作的影响比对学术和混合写作类型的影响更大。然而，干预持续时间、反馈组合和AWE平台并没有调节AWE对写作质量的影响。深入讨论了对研究和实践的影响和建议。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Educational Computing Research EDUCATION & EDUCATIONAL RESEARCH-

CiteScore

11.90

自引率

6.20%

发文量

期刊介绍： The goal of this Journal is to provide an international scholarly publication forum for peer-reviewed interdisciplinary research into the applications, effects, and implications of computer-based education. The Journal features articles useful for practitioners and theorists alike. The terms "education" and "computing" are viewed broadly. “Education” refers to the use of computer-based technologies at all levels of the formal education system, business and industry, home-schooling, lifelong learning, and unintentional learning environments. “Computing” refers to all forms of computer applications and innovations - both hardware and software. For example, this could range from mobile and ubiquitous computing to immersive 3D simulations and games to computing-enhanced virtual learning environments.