Examining Human and Automated Ratings of Elementary Students’ Writing Quality: A Multivariate Generalizability Theory Application

IF 3.5 1区教育学 Q1 EDUCATION & EDUCATIONAL RESEARCH

American Educational Research Journal Pub Date : 2022-07-08 DOI:10.3102/00028312221106773

Dandan Chen, Michael A. Hebert, Joshua Wilson

引用次数: 2

Abstract

We used multivariate generalizability theory to examine the reliability of hand-scoring and automated essay scoring (AES) and to identify how these scoring methods could be used in conjunction to optimize writing assessment. Students (n = 113) included subsamples of struggling writers and non-struggling writers in Grades 3–5 drawn from a larger study. Students wrote six essays across three genres. All essays were hand-scored by four raters and an AES system called Project Essay Grade (PEG). Both scoring methods were highly reliable, but PEG was more reliable for non-struggling students, while hand-scoring was more reliable for struggling students. We provide recommendations regarding ways of optimizing writing assessment and blending hand-scoring with AES.

查看原文本刊更多论文

小学生写作质量的人工评分与自动评分:多元概化理论的应用

我们使用多元概化理论来检验手工评分和自动作文评分(AES)的可靠性，并确定如何将这些评分方法结合使用以优化写作评估。学生(n = 113)包括从更大的研究中抽取的3-5年级挣扎作家和非挣扎作家的子样本。学生们写了三种体裁的六篇文章。所有的论文都是由四位评分者和一个名为项目论文等级(PEG)的AES系统手工评分的。两种评分方法均具有较高的信度，但PEG对非困难学生的信度更高，而手工评分对困难学生的信度更高。我们提供了关于优化写作评估和将手写评分与AES混合的方法的建议。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

American Educational Research Journal EDUCATION & EDUCATIONAL RESEARCH-

CiteScore

8.70

自引率

0.00%

发文量

期刊介绍： The American Educational Research Journal (AERJ) is the flagship journal of the American Educational Research Association, featuring articles that advance the empirical, theoretical, and methodological understanding of education and learning. It publishes original peer-reviewed analyses that span the field of education research across all subfields and disciplines and all levels of analysis. It also encourages submissions across all levels of education throughout the life span and all forms of learning. AERJ welcomes submissions of the highest quality, reflecting a wide range of perspectives, topics, contexts, and methods, including interdisciplinary and multidisciplinary work.