Three cobblers worth the mastermind? The potential of ensemble in crowdsourced classification problems

Abstract

Classification problems, where the objective is to identify the class labels of given data points, are most often the subject of open contests, in which solvers compete for awards offered by solution seekers. Extant literature in open contests has studied both the winner-takes-all and top-K award schemes, in which the award is granted to the best one and the best K solutions, respectively. However, in comparing these two schemes, researchers have never considered that under a top-K award scheme, seekers may often benefit from aggregating the solutions as an ensemble, which could achieve a performance that is superior to any individual solution. The practice of ensemble is very common especially in classification and other data science problems, and in this work, we are the first to model it with the scope of deriving the optimal award scheme in open contests. Our results formally show that, given the option of ensemble, a top-K award scheme may have the advantage to grant a higher profit to the seeker than the winner-takes-all award scheme. Further, the optimal number of awards is positively affected by contest parameters including the solvers' expertise, the number of solvers, and the return on effort, whereas it is negatively affected by the technical uncertainty when such uncertainty is sufficiently large. These findings are consistent with the increasing adoption of the top-K award scheme in contests held on Kaggle and other similar platforms.