Influence of pairing in examiner leniency and stringency ('hawk-dove effect') in part II of the European Diploma of Anaesthesiology and Intensive Care: A cohort study.
Stephen Sciberras, Markus Klimek, Bazil Ateleanu, Hugues Scipioni, Rodolphe Di Loreto, Joana Berger-Estilita
{"title":"Influence of pairing in examiner leniency and stringency ('hawk-dove effect') in part II of the European Diploma of Anaesthesiology and Intensive Care: A cohort study.","authors":"Stephen Sciberras, Markus Klimek, Bazil Ateleanu, Hugues Scipioni, Rodolphe Di Loreto, Joana Berger-Estilita","doi":"10.1097/EJA.0000000000002052","DOIUrl":null,"url":null,"abstract":"<p><strong>Background: </strong>The European Diploma of Anaesthesiology and Intensive Care (EDAIC) Part II examination is a supranational examination for anaesthesiologists.</p><p><strong>Objectives: </strong>We explore the impact of examiner pairing on leniency and stringency, commonly referred to as the 'hawk-dove effect'. We investigate the potential variations in grading approaches, resulting from different examiner pairs and their implications for candidate performance.</p><p><strong>Design: </strong>Retrospective cohort, observational design.</p><p><strong>Setting: </strong>EDAIC Part II examination data from 2021 to 2023.</p><p><strong>Participants: </strong>Three hundred and twenty-five examiners across 122 EDAIC Part II single-day examination sessions.</p><p><strong>Interventions: </strong>We analysed the influence of examiner leniency and examiner pairing on candidate performance in the EDAIC Part II using many-facet Rasch modelling.</p><p><strong>Main outcome measures: </strong>The study's main outcome measure was determining a leniency score among the examiner population. The study also aimed to assess how examiner pairing influenced candidate performance, as measured by their scores in the examination.</p><p><strong>Results: </strong>During the study period, the number of examiners who participated in 2021, 2022 and 2023 were 253, 242 and 247, respectively. The median [IQR] single-day sessions attended were 7.0 [3 to 10]. The examination data revealed a mean leniency score of 0 (95% confidence interval (CI) -0.046 to 0.046), with the standard deviation being one-third that of the candidates' ability scores. There were 1424 different pairs of examiners, with most pairs (97%) having only a one-point difference in marking. The mean leniency score for the pair of examiners was -0.053 (95% CI -0.069 to -0.037).</p><p><strong>Conclusion: </strong>The variations in grading approaches associated with different pairings emphasise the potential for the 'hawk-dove effect' to influence candidate performance and outcomes. Understanding these variations can guide curriculum development, examiner training and coupling, ensuring a balanced and equitable assessment process.</p><p><strong>Trial registration: </strong>None.</p>","PeriodicalId":11920,"journal":{"name":"European Journal of Anaesthesiology","volume":" ","pages":"921-931"},"PeriodicalIF":4.2000,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11556864/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"European Journal of Anaesthesiology","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1097/EJA.0000000000002052","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/8/28 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"ANESTHESIOLOGY","Score":null,"Total":0}
引用次数: 0
Abstract
Background: The European Diploma of Anaesthesiology and Intensive Care (EDAIC) Part II examination is a supranational examination for anaesthesiologists.
Objectives: We explore the impact of examiner pairing on leniency and stringency, commonly referred to as the 'hawk-dove effect'. We investigate the potential variations in grading approaches, resulting from different examiner pairs and their implications for candidate performance.
Setting: EDAIC Part II examination data from 2021 to 2023.
Participants: Three hundred and twenty-five examiners across 122 EDAIC Part II single-day examination sessions.
Interventions: We analysed the influence of examiner leniency and examiner pairing on candidate performance in the EDAIC Part II using many-facet Rasch modelling.
Main outcome measures: The study's main outcome measure was determining a leniency score among the examiner population. The study also aimed to assess how examiner pairing influenced candidate performance, as measured by their scores in the examination.
Results: During the study period, the number of examiners who participated in 2021, 2022 and 2023 were 253, 242 and 247, respectively. The median [IQR] single-day sessions attended were 7.0 [3 to 10]. The examination data revealed a mean leniency score of 0 (95% confidence interval (CI) -0.046 to 0.046), with the standard deviation being one-third that of the candidates' ability scores. There were 1424 different pairs of examiners, with most pairs (97%) having only a one-point difference in marking. The mean leniency score for the pair of examiners was -0.053 (95% CI -0.069 to -0.037).
Conclusion: The variations in grading approaches associated with different pairings emphasise the potential for the 'hawk-dove effect' to influence candidate performance and outcomes. Understanding these variations can guide curriculum development, examiner training and coupling, ensuring a balanced and equitable assessment process.
期刊介绍：
The European Journal of Anaesthesiology (EJA) publishes original work of high scientific quality in the field of anaesthesiology, pain, emergency medicine and intensive care. Preference is given to experimental work or clinical observation in man, and to laboratory work of clinical relevance. The journal also publishes commissioned reviews by an authority, editorials, invited commentaries, special articles, pro and con debates, and short reports (correspondences, case reports, short reports of clinical studies).
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
