Dynamic Unknown Worker Recruitment for Heterogeneous Contextual Labeling Tasks Using Adversarial Multi-Armed Bandit

Abstract

Nowadays, crowdsourcing has become an increasingly popular paradigm for large-scale data annotation. It is crucial to ensure label quality by selecting the most suitable workers for labeling tasks. Many previous works have studied the reliability of unknown workers for crowdsourcing tasks with a stochastic assumption. However, each worker's reliability varies when performing tasks with different categories. Meanwhile, the reliability of each worker is usually unknown and doesn't follow any stochastic distribution. In this paper, we propose an Adversarial multi-armed Bandit-base algorithm to handle the Unknown Worker Recruitment (ABUWR) problem without any prior stochastic assumption. In ABUWR, we determine suitable workers for each task to maximize the accumulated average accuracy of the labeling tasks under a limited budget. Specifically, $w$e model this unknown worker recruitment problem as an adversarial multi-armed bandit game and use the least confidence scheme to ensure the total accumulate accuracy. Meanwhile, $w$e theoretically prove that ABUWR has a sub-linear regret upper bound. Furthermore, we demonstrate its significant performance through extensive simulations on real-world data traces.