Bee Colony Based Worker Reliability Estimation Algorithm in Microtask Crowdsourcing

Estimation of worker reliability on microtask crowdsourcing platforms has gained attention from many researchers. On microtask platforms no worker is fully reliable for a task and it is likely that some workers are spammers, in the sense that they provide a random answer to collect the financial reward. Existence of spammers is harmful as they increase the cost of microtasking and will negatively affect the answer aggregation process. Hence, to discriminate spammers and non-spammers one needs to measure worker reliability to predict how likely that a worker put an effort in solving a task. In this paper we introduce a new reliability estimation algorithm works based on bee colony algorithm called REBECO. This algorithm relies on Gaussian process model to estimate reliability of workers dynamically. With bees that go in search of pollen, some are more successful than the others. This maps well to our problem, where some workers (i.e., bees) are more successful than other workers for a given task thus, giving rise to a reliability measure. Answer aggregation with respect to worker reliability rates has been considered as a suitable replacement for conventional majority voting. We compared REBECO with majority voting using two real world datasets. The results indicate that REBECO is able to outperform MV significantly.