Cross-worker joint modeling-based label integration for crowdsourcing

Abstract

In crowdsourcing learning, label integration algorithms are applied to infer each instance's integrated label from its multiple noisy label set. Recent advancements have demonstrated that worker modeling is an effective approach to improving label integration's performance. In real-world crowdsourced scenarios, however, each worker often annotates a few instances only, leading to insufficient worker modeling. To address this issue, we propose a novel cross-worker joint modeling-based label integration (CJMLI) algorithm. Different from existing algorithms that focus on modeling individual workers solely, CJMLI exploits cross-worker joint modeling to effectively mitigate the impact of insufficient worker modeling. Specifically, we first employ majority voting to get initial integrated labels and then apply them to estimate worker qualities. Subsequently, for each instance, we randomly select a subset of workers to estimate its class membership probabilities and then generate a weighted instance for each class. Next, we use the weighted instances to train a classifier. This process is executed several times to get multiple classifiers. Finally, we use weighted majority voting to fuse their predicted labels to infer the final integrated label of each instance. Extensive experiments demonstrate that CJMLI significantly outperforms all its competitors.