Critiquing the rationales for using comparative judgement: a call for clarity

IF 2.7 3区教育学 Q1 EDUCATION & EDUCATIONAL RESEARCH

Assessment in Education-Principles Policy & Practice Pub Date : 2022-11-02 DOI:10.1080/0969594X.2022.2147901

Katelin Kelly, M. Richardson, T. Isaacs

{"title":"Critiquing the rationales for using comparative judgement: a call for clarity","authors":"Katelin Kelly, M. Richardson, T. Isaacs","doi":"10.1080/0969594X.2022.2147901","DOIUrl":null,"url":null,"abstract":"ABSTRACT Comparative judgment is gaining popularity as an assessment tool, including for high-stakes testing purposes, despite relatively little research on the use of the technique. Advocates claim two main rationales for its use: that comparative judgment is valid because humans are better at comparative than absolute judgment, and because it distils the aggregate view of expert judges. We explore these contentions. We argue that the psychological underpinnings used to justify the method are superficially treated in the literature. We conceptualise and critique the notion that comparative judgment is ‘intrinsically valid’ due to its use of expert judges. We conclude that the rationales as presented by the comparative judgment literature are incomplete and inconsistent. We recommend that future work should clarify its position regarding the psychological underpinnings of comparative judgment, and if necessary present a more compelling case; for example, by integrating the comparative judgment literature with evidence from other fields.","PeriodicalId":51515,"journal":{"name":"Assessment in Education-Principles Policy & Practice","volume":null,"pages":null},"PeriodicalIF":2.7000,"publicationDate":"2022-11-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"3","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Assessment in Education-Principles Policy & Practice","FirstCategoryId":"95","ListUrlMain":"https://doi.org/10.1080/0969594X.2022.2147901","RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 3

Abstract

ABSTRACT Comparative judgment is gaining popularity as an assessment tool, including for high-stakes testing purposes, despite relatively little research on the use of the technique. Advocates claim two main rationales for its use: that comparative judgment is valid because humans are better at comparative than absolute judgment, and because it distils the aggregate view of expert judges. We explore these contentions. We argue that the psychological underpinnings used to justify the method are superficially treated in the literature. We conceptualise and critique the notion that comparative judgment is ‘intrinsically valid’ due to its use of expert judges. We conclude that the rationales as presented by the comparative judgment literature are incomplete and inconsistent. We recommend that future work should clarify its position regarding the psychological underpinnings of comparative judgment, and if necessary present a more compelling case; for example, by integrating the comparative judgment literature with evidence from other fields.

查看原文本刊更多论文

批判使用比较判断的基本原理:要求澄清

比较判断作为一种评估工具越来越受欢迎，包括用于高风险的测试目的，尽管对该技术的使用研究相对较少。提倡者声称，使用这种方法有两个主要理由:一是比较判断是有效的，因为人类更擅长比较判断，而不是绝对判断;二是它提炼了专家判断的总体观点。我们将探讨这些争论。我们认为，用于证明该方法的心理基础在文献中被肤浅地对待。我们概念化和批判比较判断由于使用专家法官而具有“内在有效性”的概念。我们的结论是，比较判断文献提出的基本原理是不完整和不一致的。我们建议未来的工作应该澄清其关于比较判断的心理基础的立场，并在必要时提出一个更有说服力的案例;例如，通过将比较判断文献与其他领域的证据相结合。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Assessment in Education-Principles Policy & Practice EDUCATION & EDUCATIONAL RESEARCH-

CiteScore

5.70

自引率

3.10%

发文量

期刊介绍： Recent decades have witnessed significant developments in the field of educational assessment. New approaches to the assessment of student achievement have been complemented by the increasing prominence of educational assessment as a policy issue. In particular, there has been a growth of interest in modes of assessment that promote, as well as measure, standards and quality. These have profound implications for individual learners, institutions and the educational system itself. Assessment in Education provides a focus for scholarly output in the field of assessment. The journal is explicitly international in focus and encourages contributions from a wide range of assessment systems and cultures. The journal''s intention is to explore both commonalities and differences in policy and practice.