Learning from the Crowd with Neural Network

Abstract

In general, the first step for supervised learning from crowdsourced data is integration. To obtain training data as traditional machine learning, the ground truth for each example in the crowdsourcing dataset must be integrated with consensus algorithms. However, some information and correlations among labels in the crowdsourcing dataset have discarded after integration. In order to study whether the information and correlations are useful for learning, we proposed three types of neural networks. Experimental results show that i) all the three types of neural networks have abilities to predict labels for future unseen examples, ii) when labelers have lower qualities, the information and correlations in crowdsourcing datasets, which are discarded by integration, does improve the performance of neural networks significantly, iii) when labelers have higher label qualities, the information and correlations have little impact on improving accuracy of neural networks.