Hierarchical cooperation of experts in learning from crowds

Abstract

Crowdsourcing allows us to utilize non-expert annotators in learning concept instead of using reference model. In machine learning domain, there are many papers which addressed this problem by assuming independency between annotators. While this assumption does not hold in real world's problems. In this paper we propose a hierarchical framework to model dependency between annotators. This cooperation of experts make the predicted model robust to deviation from ground-truth. Parameters are obtained using maximum likelihood estimator with an iterative EM algorithm. The mathematical derivations indicate that the precision of a follower annotator depends on precision of its followee expert. Experimental results on synthetic, UCI and MNIST datasets show superiority of the proposed algorithm in comparison with its competitors.