Evaluating Differential Rater Accuracy Over Time in Solo Music Performance Assessment

Abstract

ABSTR ACT In formal music performance assessments where raters evaluate performances by different students or ensembles over several days, the quality of raters’ judgments can vary over the duration of the assessment. Because time is construct-irrelevant, the influence of time on rater judgments poses a threat to the fairness of the assessment. In this study, we used Rasch measurement theory to explore changes in rater accuracy over time in the context of a secondary-level solo instrumental music performance assessment, where rater accuracy is defined as the match between observed ratings and criterion ratings. Specifically, we considered changes in rater accuracy related to the overall group of raters, individual raters, and rubric domains. Results suggested that the raters became less accurate over time, and there were differences in rater accuracy within domains over time. Together, our findings highlight the dynamic nature of rater accuracy over time. Our findings also highlight the importance of empirically examining changes in rater accuracy over time using the approach illustrated in this study. Implications for investigating rater accuracy over time in both research-based and applied assessment contexts are discussed.