Productive teaming under uncertainty: when a human and a machine classify objects together

Abstract

We study the task of object categorization in an industrial setting. Typically, a machine classifies objects according to an internal, inferred model, and calls to a human worker if it is uncertain. However, the human worker may be also uncertain. We elaborate on the challenges and solutions to assess the certainty of the human without disturbing the industrial process, and to assess label reliability and human certainty in conventional object classification and crowdworking. Albeit there are methods for measuring stress, insights on the correlation of stress and uncertainty and uncertainty indicators during labeling by humans, these advances are yet to be combined to solve the aforementioned uncertainty challenge. We propose a solution as a sequence of tasks, starting with a experiment that measures human certainty in a task of controlled difficulty, whereupon we can associate certainty with correctness and levels of vital signals.