Cost models of Quality assurance in crowdsourcing

Abstract

Even though crowdsourcing was introduced as a novel business model for risk prone entrepreneurs, in time it has become a valid means of producing value in large scale projects as well. However due to loose employee - worker relationship, quality continues to be a serious issue for crowdsourcing. Therefore, various quality assurance mechanisms are used in crowdsourcing. Quality assurance significantly impacts project costs. However the literature lacks defined ways to estimate costs of quality and guidelines for selecting the most cost-effective mechanism. In this paper we introduce cost models, which may be used to estimate crowdsourcing costs and act as a guideline to efficiently utilize quality assurance mechanisms. These models were developed through a Cost of Quality (CoQ) approach, which has been widely used in total quality management since the 1970's. To demonstrate the applicability of our approach, we present the results of an experiment in which cost models are applied on subjective, non-deterministic microtask type crowdsourcing.