E2EPref: An end-to-end preference-based framework for speech quality assessment to alleviate bias in direct assessment scores

Abstract

In speech quality assessment (SQA), direct assessment (DA) scores are frequently used as the objective of model training. However, because the DA scores themselves have listener-wise bias and equal range bias, the scores predicted by models trained with DA scores do not always reflect the true quality score. In this study, we utilize preference-based learning for SQA by transforming the DA score prediction framework into a preference prediction framework. Our proposed End-to-End Preference-based framework (E2EPref) for SQA is designed for predicting system-level quality scores directly. It contains four proposed components: pair generation, preference function, threshold selection, and preference aggregation. Through these functions of E2EPref, we aim to mitigate biases introduced by directly using DA scores for training. In experiments, we show that this framework helps the SQA model alleviate biases, resulting in higher system-level Spearman’s rank correlation coefficient and linear correlation coefficient. Additionally, we evaluate the quality prediction capability of the framework in a zero-shot out-of-domain scenario. Finally, we collect subjective preference scores on a dataset already containing DA scores and analyze the advantages and disadvantages of using DA scores versus subjective preference scores as the ground truth or for model training.