Modeling rater judgments of interpreting quality: Ordinal logistic regression using neural-based evaluation metrics, acoustic fluency measures, and computational linguistic indices

Abstract

Human raters remain central to interpreting quality assessment (IQA); however, recent years have witnessed a growing body of research exploring automatic assessment. These studies have used machine translation evaluation metrics, acoustic fluency measures, and computational linguistic indices as separate approaches to model rater judgments of interpreting quality. Nonetheless, limited research has integrated these three types of measures within a single framework. To address this gap, this exploratory and proof-of-concept study adopts an integrative approach, combining these three types of measures to model rater judgments of interpreting quality as a classification problem. Using a dataset of 161 Chinese-to-English interpretations, we applied ordinal logistic regression analysis to identify significant predictors across fidelity, fluency, and linguistic dimensions. The analyses yielded two sets of significant predictors, including (a) COMET-22, mean length of unfilled pauses, mean length of run, and mean word length, and (b) BLEURT-20, phonation time ratio, speech rate, mean word length, type-token ratio for content words, type-token ratio for all words, and mean word frequency for content words. These models performed well on the testing dataset, particularly for classifying interpretations into four bands of overall interpreting quality (e.g., accuracy = .643, 1-off accuracy = .805), based on the Rasch-calibrated scores from human evaluation. These findings suggest that this integrated approach may enhance the precision and scalability of IQA and has the potential to reduce logistical burdens in large-scale professional interpreter certification exams and language proficiency tests.