Automatic design optimization of preference-based subjective evaluation with online learning in crowdsourcing environment

Abstract

Preference-based subjective evaluation is a key method for reliably evaluating generative media. However, its huge number of pair combinations makes it prohibitively difficult to apply to large-scale evaluation using crowdsourcing. To address this issue, we propose an automatic optimization method for preference-based subjective evaluation in terms of pair combination selections and the allocation of evaluation volumes with online learning in a crowdsourcing environment. We use a preference-based online learning method based on a sorting algorithm to identify the total order of systems with minimum sample volumes. Our online learning algorithm supports parallel and asynchronous executions under fixed-budget conditions required for crowdsourcing. Our experiment on the preference-based subjective evaluation of synthetic speech on naturalness shows that our method successfully optimizes the preference-based test by reducing the number of pair combinations from 351 to 83 and allocating optimal evaluation volumes for each pair ranging from 30 to 663 without compromising evaluation errors and wasting budget allocations.