Crowdsourcing System for Numerical Tasks based on Latent Topic Aware Worker Reliability

Abstract

Crowdsourcing is a widely adopted way for various labor-intensive tasks. One of the core problems in crowdsourcing systems is how to assign tasks to most suitable workers for better results, which heavily relies on the accurate profiling of each worker’s reliability for different topics of tasks. Many previous work have studied worker reliability for either explicit topics represented by task descriptions or latent topics for categorical tasks. In this work, we aim to accurately estimate more fine-grained worker reliability for latent topics in numerical tasks, so as to further improve the result quality. We propose a bayesian probabilistic model named Gaussian Latent Topic Model(GLTM) to mine the latent topics of numerical tasks based on workers’ behaviors and to estimate workers’ topic-level reliability. By utilizing the GLTM, we propose a truth inference algorithm named TI-GLTM to accurately infer the tasks’ truth and topics simultaneously and dynamically update workers’ topic-level reliability. We also design an online task assignment mechanism called MRA-GLTM, which assigns appropriate tasks to workers with the Maximum Reduced Ambiguity principle. The experiment results show our algorithms can achieve significantly lower MAE and MSE than that of the state-of-the-art approaches.