Is the use of multiple-choice items and a holistically-scored paragraph translation task fair? Examining a large-scale translation subtest

IF 2.3 3区教育学 Q1 EDUCATION & EDUCATIONAL RESEARCH

Asia Pacific Education Review Pub Date : 2024-08-04 DOI:10.1007/s12564-024-09993-y

Zhiqiang Yang, Chengyuan Yu

{"title":"Is the use of multiple-choice items and a holistically-scored paragraph translation task fair? Examining a large-scale translation subtest","authors":"Zhiqiang Yang, Chengyuan Yu","doi":"10.1007/s12564-024-09993-y","DOIUrl":null,"url":null,"abstract":"<div><p>This study investigated the test fairness of the translation section of a large-scale English test in China by examining its Differential Test Functioning (DTF) and Differential Item Functioning (DIF) across gender and major. Regarding DTF, the entire translation section exhibits partial strong measurement invariance across female and male test takers, while exhibiting full measurement invariance across test takers in (1) arts & humanities and social sciences (A&HSS) and (2) science, technology, engineering or mathematics (STEM) majors. No major-based DIF was detected in this study. Objective test items tend to favor male test takers, while direct translation test task was more favorable to females. Combining the DIF and DTF results, there may be a cancelation effect in our case. However, the effect size of DIF is either negligible or slight to moderate, indicating minimal impact on the overall fairness of the translation test task. This study further discusses the necessity of exploring the source of DIF and the importance of combining DIF and DTF for test fairness research.</p></div>","PeriodicalId":47344,"journal":{"name":"Asia Pacific Education Review","volume":"26 2","pages":"493 - 502"},"PeriodicalIF":2.3000,"publicationDate":"2024-08-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://link.springer.com/content/pdf/10.1007/s12564-024-09993-y.pdf","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Asia Pacific Education Review","FirstCategoryId":"95","ListUrlMain":"https://link.springer.com/article/10.1007/s12564-024-09993-y","RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 0

Abstract

This study investigated the test fairness of the translation section of a large-scale English test in China by examining its Differential Test Functioning (DTF) and Differential Item Functioning (DIF) across gender and major. Regarding DTF, the entire translation section exhibits partial strong measurement invariance across female and male test takers, while exhibiting full measurement invariance across test takers in (1) arts & humanities and social sciences (A&HSS) and (2) science, technology, engineering or mathematics (STEM) majors. No major-based DIF was detected in this study. Objective test items tend to favor male test takers, while direct translation test task was more favorable to females. Combining the DIF and DTF results, there may be a cancelation effect in our case. However, the effect size of DIF is either negligible or slight to moderate, indicating minimal impact on the overall fairness of the translation test task. This study further discusses the necessity of exploring the source of DIF and the importance of combining DIF and DTF for test fairness research.

查看原文本刊更多论文

选择题和综合评分段落翻译任务的使用公平吗？检查大规模翻译子测试

本研究通过考察翻译部分在性别和专业上的差异测试功能（DTF）和差异项目功能（DIF），探讨了中国大规模英语考试翻译部分的考试公平性。关于DTF，整个翻译部分在女性和男性考生中表现出部分强测量不变性，而在(1)艺术与人文社会科学（A&；HSS）和(2)科学、技术、工程或数学（STEM）专业的考生中表现出完全的测量不变性。本研究未检测到基于major的DIF。客观测试项目倾向于男性考生，而直接翻译测试任务更倾向于女性考生。结合DIF和DTF结果，在我们的案例中可能存在抵消效应。然而，DIF的效应大小可以忽略不计或轻微到中等，表明对翻译测试任务的整体公平性的影响最小。本文进一步探讨了探究DIF来源的必要性，以及将DIF与DTF结合起来进行考试公平性研究的重要性。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Asia Pacific Education Review EDUCATION & EDUCATIONAL RESEARCH-

CiteScore

5.20

自引率

4.30%

发文量

期刊介绍： The Asia Pacific Education Review (APER) aims to stimulate research, encourage academic exchange, and enhance the professional development of scholars and other researchers who are interested in educational and cultural issues in the Asia Pacific region. APER covers all areas of educational research, with a focus on cross-cultural, comparative and other studies with a broad Asia-Pacific context. APER is a peer reviewed journal produced by the Education Research Institute at Seoul National University. It was founded by the Institute of Asia Pacific Education Development, Seoul National University in 2000, which is owned and operated by Education Research Institute at Seoul National University since 2003. APER requires all submitted manuscripts to follow the seventh edition of the Publication Manual of the American Psychological Association (APA; http://www.apastyle.org/index.aspx).