An intelligent crowd-worker selection approach for reliable content labeling of food images

Abstract

Food journaling is an effective way to regulate excessive food intake. However manual food journaling is burdensome, and crowd-assisted food journaling has been explored to ease user burden. The crowd-assisted journaling uses a label & verify approach where an end-user uploads his/her food image and paid crowd-workers label content of the image. Then another set of crowd-workers verify the labels for correctness. In this paper, we propose an alternative approach where we label food images with only high performing labelers. Since high performing labelers generally provide good quality labels, our approach achieves high accuracy without verifying the food labels for correctness. We also propose a machine learning algorithm to automatically identify high performing crowd-labelers from a dataset of 3925 images collected over 5 months. Such automated identification of high performing workers and elimination of needless verification reduce cost of food labeling. Specially for large scale deployments where large number of images need to be labeled, our approach can reduce overall expenses by half.