Predicting result quality in Crowdsourcing using application layer monitoring

Abstract

Crowdsourcing has become a valuable tool for many business applications requiring to meet a certain quality of the results generated by the workers. Therefore, several quality assurance mechanisms have been developed which are partly deployed in commercial crowdsourcing platforms. However, these mechanisms usually impose additional work overhead for the worker, e.g. by adding test questions, or increase the costs for the employer, e.g. by replicating the task for majority decisions. In this work, we analyze the applicability of implicit measurements to objectively estimate the quality of the workers' results. First efforts in this area have already been made by investigating the impact of the task completion time. We extend this research by deploying an application layer monitoring (ALM), which enables monitoring the workers' interactions with our task interface on a much more detailed level. Based on an exemplary use case, we discuss a possible implementation and demonstrate the potential of the approach by predicting the quality of the workers' submission based on our monitoring results. This ALM provides a new way to identify low quality work as well as difficulties in fulfilling the formulated tasks in the domain of Crowdsourcing.