Return-of-Interest Conscious Truth Inference for Crowdsourcing Queries

In the big data era, the flourishing development of Internet services brings a lot of user generated data, in which most new information cannot be systematically retrieved by current knowledge bases. For example, a dramatic number of new hashtags appear in the social media every day, resulting in much unknown but valuable knowledge that requires reliable category/attribute labeling strategies. The crowdsourcing platform provides an effective tool to leverage opinions from the Internet crowd. In this paper, we propose incorporating varied task importance, called Return of Interest (RoI), into resource allocation in crowdsourcing. The awareness of RoI is important in the business sense, but it introduces new challenges. In this paper, we propose a two-phase framework, called Macro-Assignment and Micro-Optimization (MAMO), to simultaneously consider the issue of budget allocation and the chance of iteratively obtaining RoI. With the fixed budget, we prove that worker allocation to diverse pools for the best expectation of RoI in return is a NPhard challenge. We propose a Dynamic-Programming strategy to resolve the issue effectively. As shown in our experimental results, we demonstrate that the DP-based strategy can significantly outperform the baseline greedy approaches, also indicating its feasibility to be deployed as the standard component for budget allocation in crowdsourcing.