Efficient Worker Selection Through History-Based Learning in Crowdsourcing

Abstract

Crowdsourcing has emerged as a promising approach for obtaining services and data in a short time and at a reasonable budget. However, the quality of the output provided by the crowd is not guaranteed, and must be controlled. This quality control usually relies on worker screening or contribution reviewing at the cost of additional time and budget overheads. In this paper, we propose to reduce these overheads by leveraging the system history. We describe an offline learning algorithm that groups tasks from history into homogeneous clusters and learns for each cluster the worker features that optimize the contribution quality. These features are then used by the online targeting algorithm to select reliable workers for each incoming task. The proposed method is compared to the state of the art selection methods using real world datasets. Results show that we achieve comparable, and in some cases better, output quality for a smaller budget and shorter time.