A robust inference algorithm for crowd sourced categorization

Abstract

With the rapid growing of crowdsourcing systems, class labels for supervised learning can be easily obtained from crowdsourcing platforms. To deal with the problem that labels obtained from crowds are usually noisy due to imperfect reliability of non-expert workers, we let multiple workers provide labels for the same object. Then, true labels of the labeled object are estimated through ground truth inference algorithms. The inferred integrated labels are expected to be of high quality. In this paper, we propose a novel ground truth inference algorithm based on EM algorithm, which not only infers the true labels of the instances but also simultaneously estimates the reliability of each worker and the difficulty of each instance. Experimental results on seven real-world crowdsourcing datasets show that our proposed algorithm outperforms eight state-of-the art algorithms.