小学生写作质量的人工评分与自动评分:多元概化理论的应用

IF 3.5 1区教育学 Q1 EDUCATION & EDUCATIONAL RESEARCH

American Educational Research Journal Pub Date : 2022-07-08 DOI:10.3102/00028312221106773

Dandan Chen, Michael A. Hebert, Joshua Wilson

{"title":"小学生写作质量的人工评分与自动评分:多元概化理论的应用","authors":"Dandan Chen, Michael A. Hebert, Joshua Wilson","doi":"10.3102/00028312221106773","DOIUrl":null,"url":null,"abstract":"We used multivariate generalizability theory to examine the reliability of hand-scoring and automated essay scoring (AES) and to identify how these scoring methods could be used in conjunction to optimize writing assessment. Students (n = 113) included subsamples of struggling writers and non-struggling writers in Grades 3–5 drawn from a larger study. Students wrote six essays across three genres. All essays were hand-scored by four raters and an AES system called Project Essay Grade (PEG). Both scoring methods were highly reliable, but PEG was more reliable for non-struggling students, while hand-scoring was more reliable for struggling students. We provide recommendations regarding ways of optimizing writing assessment and blending hand-scoring with AES.","PeriodicalId":48375,"journal":{"name":"American Educational Research Journal","volume":"1 1","pages":"1122 - 1156"},"PeriodicalIF":3.5000,"publicationDate":"2022-07-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"2","resultStr":"{\"title\":\"Examining Human and Automated Ratings of Elementary Students’ Writing Quality: A Multivariate Generalizability Theory Application\",\"authors\":\"Dandan Chen, Michael A. Hebert, Joshua Wilson\",\"doi\":\"10.3102/00028312221106773\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"We used multivariate generalizability theory to examine the reliability of hand-scoring and automated essay scoring (AES) and to identify how these scoring methods could be used in conjunction to optimize writing assessment. Students (n = 113) included subsamples of struggling writers and non-struggling writers in Grades 3–5 drawn from a larger study. Students wrote six essays across three genres. All essays were hand-scored by four raters and an AES system called Project Essay Grade (PEG). Both scoring methods were highly reliable, but PEG was more reliable for non-struggling students, while hand-scoring was more reliable for struggling students. We provide recommendations regarding ways of optimizing writing assessment and blending hand-scoring with AES.\",\"PeriodicalId\":48375,\"journal\":{\"name\":\"American Educational Research Journal\",\"volume\":\"1 1\",\"pages\":\"1122 - 1156\"},\"PeriodicalIF\":3.5000,\"publicationDate\":\"2022-07-08\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"2\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"American Educational Research Journal\",\"FirstCategoryId\":\"95\",\"ListUrlMain\":\"https://doi.org/10.3102/00028312221106773\",\"RegionNum\":1,\"RegionCategory\":\"教育学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"EDUCATION & EDUCATIONAL RESEARCH\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"American Educational Research Journal","FirstCategoryId":"95","ListUrlMain":"https://doi.org/10.3102/00028312221106773","RegionNum":1,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 2

摘要

我们使用多元概化理论来检验手工评分和自动作文评分(AES)的可靠性，并确定如何将这些评分方法结合使用以优化写作评估。学生(n = 113)包括从更大的研究中抽取的3-5年级挣扎作家和非挣扎作家的子样本。学生们写了三种体裁的六篇文章。所有的论文都是由四位评分者和一个名为项目论文等级(PEG)的AES系统手工评分的。两种评分方法均具有较高的信度，但PEG对非困难学生的信度更高，而手工评分对困难学生的信度更高。我们提供了关于优化写作评估和将手写评分与AES混合的方法的建议。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

Examining Human and Automated Ratings of Elementary Students’ Writing Quality: A Multivariate Generalizability Theory Application

We used multivariate generalizability theory to examine the reliability of hand-scoring and automated essay scoring (AES) and to identify how these scoring methods could be used in conjunction to optimize writing assessment. Students (n = 113) included subsamples of struggling writers and non-struggling writers in Grades 3–5 drawn from a larger study. Students wrote six essays across three genres. All essays were hand-scored by four raters and an AES system called Project Essay Grade (PEG). Both scoring methods were highly reliable, but PEG was more reliable for non-struggling students, while hand-scoring was more reliable for struggling students. We provide recommendations regarding ways of optimizing writing assessment and blending hand-scoring with AES.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

American Educational Research Journal EDUCATION & EDUCATIONAL RESEARCH-

CiteScore

8.70

自引率

0.00%

发文量

期刊介绍： The American Educational Research Journal (AERJ) is the flagship journal of the American Educational Research Association, featuring articles that advance the empirical, theoretical, and methodological understanding of education and learning. It publishes original peer-reviewed analyses that span the field of education research across all subfields and disciplines and all levels of analysis. It also encourages submissions across all levels of education throughout the life span and all forms of learning. AERJ welcomes submissions of the highest quality, reflecting a wide range of perspectives, topics, contexts, and methods, including interdisciplinary and multidisciplinary work.