All types of experience are equal, but some are more equal: The effect of different types of experience on rater severity and rater consistency

IF 2.2 1区文学 0 LANGUAGE & LINGUISTICS

Language Testing Pub Date : 2024-04-10 DOI:10.1177/02655322241239362

Reeta Neittaanmäki, Iasonas Lamprianou

{"title":"All types of experience are equal, but some are more equal: The effect of different types of experience on rater severity and rater consistency","authors":"Reeta Neittaanmäki, Iasonas Lamprianou","doi":"10.1177/02655322241239362","DOIUrl":null,"url":null,"abstract":"This article focuses on rater severity and consistency and their relation to different types of rater experience over a long period of time. The article is based on longitudinal data collected from 2009 to 2019 from the second language Finnish speaking subtest in the National Certificates of Language Proficiency in Finland. The study investigated whether rater severity and consistency are affected differently by different types of rater experience and by skipping rating sessions. The data consisted of 45 rating sessions with 104 raters and 59,899 examinees and were analyzed using the Many-Facets Rasch model and generalized linear mixed models. The results showed that when the raters gained more rating experience, they became slightly more lenient, but different types of experience had quantitatively different magnitudes of impact. In addition, skipping rating sessions, and in that way disconnecting from the rater community, increased the likelihood of a rater to be inconsistent. Finally, we provide methodological recommendations for future research and consider implications for practice.","PeriodicalId":17928,"journal":{"name":"Language Testing","volume":"25 1","pages":""},"PeriodicalIF":2.2000,"publicationDate":"2024-04-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Language Testing","FirstCategoryId":"98","ListUrlMain":"https://doi.org/10.1177/02655322241239362","RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"0","JCRName":"LANGUAGE & LINGUISTICS","Score":null,"Total":0}

引用次数: 0

Abstract

This article focuses on rater severity and consistency and their relation to different types of rater experience over a long period of time. The article is based on longitudinal data collected from 2009 to 2019 from the second language Finnish speaking subtest in the National Certificates of Language Proficiency in Finland. The study investigated whether rater severity and consistency are affected differently by different types of rater experience and by skipping rating sessions. The data consisted of 45 rating sessions with 104 raters and 59,899 examinees and were analyzed using the Many-Facets Rasch model and generalized linear mixed models. The results showed that when the raters gained more rating experience, they became slightly more lenient, but different types of experience had quantitatively different magnitudes of impact. In addition, skipping rating sessions, and in that way disconnecting from the rater community, increased the likelihood of a rater to be inconsistent. Finally, we provide methodological recommendations for future research and consider implications for practice.

查看原文本刊更多论文

所有类型的经验都是平等的，但有些经验更为平等：不同类型的经验对评分者严重程度和评分者一致性的影响

本文重点研究了评分者的严重性和一致性，以及它们与不同类型的评分者长期经验之间的关系。文章基于 2009 年至 2019 年期间从芬兰国家语言能力证书第二语言芬兰语口语子测试中收集的纵向数据。研究调查了不同类型的评分者经验和跳过评分环节是否会对评分者的严重程度和一致性产生不同影响。数据由 104 名评分员和 59,899 名受试者的 45 个评分环节组成，并使用 Many-Facets Rasch 模型和广义线性混合模型进行了分析。结果表明，当评分者获得更多的评分经验时，他们会变得稍微宽松一些，但不同类型的经验会产生数量上不同的影响。此外，跳过评分环节，从而脱离评分者群体，会增加评分者不一致的可能性。最后，我们为今后的研究提供了方法建议，并考虑了对实践的影响。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Language Testing Multiple-

CiteScore

6.70

自引率

9.80%

发文量

期刊介绍： Language Testing is a fully peer reviewed international journal that publishes original research and review articles on language testing and assessment. It provides a forum for the exchange of ideas and information between people working in the fields of first and second language testing and assessment. This includes researchers and practitioners in EFL and ESL testing, and assessment in child language acquisition and language pathology. In addition, special attention is focused on issues of testing theory, experimental investigations, and the following up of practical implications.