Effects of raters’ professional backgrounds on assessing interpreting quality: An exploratory mixed-methods investigation into rater behavior

Abstract

Rater effects have received long-term scholarly attention in language performance testing. Presumably, distinct rater effect is indispensable from raters' language and professional backgrounds. However, to the best of our knowledge, the current empirical literature on interpreting testing and assessment (ITA) has been under scarce scrutiny of the relevance between rater behavior and varied professional backgrounds. Drawing on the framework of language teacher cognition, the thread of this exploratory study followed a mixed-methods design to investigate behaviors of raters (N = 9) in the context of ITA. The multi-faceted Rasch model (MFRM) yielded quantitative results regarding rater behavior. Notably, five raters who exhibited abnormal rater behaviors, including extreme severity/leniency, self-(in)consistency, and biased ratings, were selected to participate in a stimulated recall. The qualitative insights helped elucidate the connection between inconsistent rater behaviors and their professional backgrounds. The study identified professional backgrounds as a factor contributing to teacher cognition about their assessment behavior. These findings encourage two wide-open avenues for the AI-augmented ITA research across different raters' identities and the research on raters’ digital competence of using AI in ITA.