Going beyond hawks and doves - Measuring degrees of examiner misalignment in OSCEs.

Abstract

Background: Minimising examiner differences in scoring in OSCEs is key in supporting the validity of the assessment outcomes. This is particularly true for common OSCE designs where the same station is administered across parallel circuits, with examiners nested within these. However, the common classification of extreme examiners as 'hawks' or 'doves' can be overly simplistic. Rather, it is the difference in patterns of scoring across circuits that better indicates poor levels of agreement between examiners that can unfairly advantage particular groups of candidates in comparison with others in different circuits.

Methods and materials: In this paper, a new measure of differences in examiner scoring is presented that quantifies the different combined patterns of scoring in global grades and checklist/domain scores for pairs of examiners assessing in the same station but in different circuits. Based on calculating the area between separate examiners' individual borderline regression lines, this measure can be used as a post hoc metric to provide a broad range of validity evidence for the assessment and its outcomes.

Results and conclusions: In challenging the 'hawks'/'doves' paradigm, this work presents a detailed empirical analysis of a new misalignment metric in a particular high-stakes context and gives a range of evidence of its contribution to overall OSCE quality control processes and of improved fairness to candidates over time. The paper concludes with comments on developing the metric to contexts where there are multiple parallel circuits which will allows its practical application to a broader set of OSCE contexts.