Integrating Active Learning with Supervision for Crowdsourcing Generalization

With various online crowdsourcing platforms, it is easy to collect multiple labels for the same examples from the crowd. Consensus integration algorithms can infer the estimated ground truths from the multiple label sets of these crowdsourcing datasets. However, it couldn't be avoided that these integrated (estimated) labels still contain noises. In order to further improve the performance of a model learned from data with these integrated labels, we propose an active learning framework to further improve the data quality, such that to improve the model quality, through acquiring limited true labels from experts (the oracle). We further investigate two active learning strategies in terms of two uncertainty measures (i.e., CLUE and MUE) within the active learning framework. From our experimental results on eight simulation crowdsourcing datasets and four real-world crowdsourcing datasets with three popular consensus integration algorithms, we draw several conclusions as follows. (i) Our active learning framework with the input from the oracle significantly improves the generalization ability of the model learned from crowdsourcing data. (ii) Our two active learning strategies outperform a random active learning strategy.