Adaptive Selection of Working Conditions for Crowdsourced Tasks

This paper proposes a method of working condition selection based on type identification of crowd workers. Here, the working condition selection means finding the values of working conditions that are suitable for individual workers. Multi-armed bandit techniques are promising, but it may happen that exploring various task settings for a single worker interferes with that worker, which deteriorates the quality of contributions. To solve this problem, we introduce the type identification test, i.e., we divide the entire period for a worker into a type identification phase and an execution phase and alternately handle the calculation at the individual level and at the aggregate level. Our method can find an appropriate task setting without exploring various settings for a worker, i.e., excessively interfering with the worker. Also, we provide a method of calculating the optimal type identification test to maximize the expected quality of contributions in the execution phase. Finally, we show our method outperforms conventional multi-armed bandit algorithms such as Softmax and UCB1 with data we collected on the Amazon Mechanical Turk and with a simulation.