{"title":"Comparing computer-based and paper-based rating modes in an English writing test","authors":"Yuhua Liu , Jianda Liu","doi":"10.1016/j.asw.2023.100771","DOIUrl":null,"url":null,"abstract":"<div><p>The study utilized a mixed methods approach to compare the scoring of raters in assessing writing performance across three modes: paper-based, on-screen marking of scanned images, and online word-processed versions. six experienced raters evaluated the performances of 39 test-takers in each mode. The many-facet Rasch model was employed to analyze scoring differences among the rating modes; the semi-structured interview was used to collect raters' perceptions towards performance under the three modes. The findings indicated that the difficulty level was ranked in ascending order of on-screen marking of scanned images, paper-based text, and online word-processed text. Bias analysis revealed interactions between the rater and the mode, as well as between the criterion and the mode. Verbal reports from the raters highlighted four construct-irrelevant factors that could potentially influence scoring under the three modes: convenience for essay overview and word recognition, potential underestimation of word count, and raters' preference for essays in handwriting. Based on the results, recommendations were provided for rater training and essay scoring across different modes.</p></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":null,"pages":null},"PeriodicalIF":4.2000,"publicationDate":"2023-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Assessing Writing","FirstCategoryId":"98","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S107529352300079X","RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}
引用次数: 0
Abstract
The study utilized a mixed methods approach to compare the scoring of raters in assessing writing performance across three modes: paper-based, on-screen marking of scanned images, and online word-processed versions. six experienced raters evaluated the performances of 39 test-takers in each mode. The many-facet Rasch model was employed to analyze scoring differences among the rating modes; the semi-structured interview was used to collect raters' perceptions towards performance under the three modes. The findings indicated that the difficulty level was ranked in ascending order of on-screen marking of scanned images, paper-based text, and online word-processed text. Bias analysis revealed interactions between the rater and the mode, as well as between the criterion and the mode. Verbal reports from the raters highlighted four construct-irrelevant factors that could potentially influence scoring under the three modes: convenience for essay overview and word recognition, potential underestimation of word count, and raters' preference for essays in handwriting. Based on the results, recommendations were provided for rater training and essay scoring across different modes.
期刊介绍：
Assessing Writing is a refereed international journal providing a forum for ideas, research and practice on the assessment of written language. Assessing Writing publishes articles, book reviews, conference reports, and academic exchanges concerning writing assessments of all kinds, including traditional (direct and standardised forms of) testing of writing, alternative performance assessments (such as portfolios), workplace sampling and classroom assessment. The journal focuses on all stages of the writing assessment process, including needs evaluation, assessment creation, implementation, and validation, and test development.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
