From black box to transparency: Enhancing automated interpreting assessment with explainable AI in college classrooms

Abstract

Recent advancements in machine learning have spurred growing interests in automated interpreting quality assessment. Nevertheless, existing research is subject to certain limitations, including the insufficient examination of language use quality, restricted modeling effectiveness due to data scarcity at the highest and lowest performance tiers, and a lack of efforts to explain model predictions. To address these gaps, the present study proposes a multi-dimensional modeling framework that integrates feature engineering, data augmentation, and explainable machine learning. This approach prioritizes explainability over “black box” predictions by utilizing only construct-relevant, transparent features and conducting SHAP analysis, an explainable AI (XAI) method. Our results demonstrated relatively strong predictive performance on a self-compiled English-Chinese consecutive interpreting dataset: XGBoost excelled in predicting fluency (ρ = 0.86, RMSE = 0.61) and target language use (ρ = 0.79, RMSE = 0.75), while Random Forest was optimal for modeling information completeness (ρ = 0.68, RMSE = 1.05). SHAP analysis identified the strongest predictive features for each dimension: BLEURT and CometKiwi scores for information completeness, pause-related features for fluency, and Chinese-specific phraseological diversity metrics for language use. Overall, this study presents a scalable, reliable, and transparent alternative to traditional human evaluation, holding significant implications for automated language assessment. Notably, the emphasis on explainability facilitates the provision of detailed diagnostic feedback for learners and supports self-regulated learning—advantages not afforded by automated scores in isolation.