Active learning from noisy and abstention feedback

Abstract

An active learner is given an instance space, a label space and a hypothesis class, where one of the hypotheses in the class assigns ground truth labels to instances. Additionally, the learner has access to a labeling oracle, which it can interactively query for the label of any example in the instance space. The goal of the learner is to find a good estimate of the hypothesis in the hypothesis class that generates the ground truth labels while making as few interactive queries to the oracle as possible. This work considers a more general setting where the labeling oracle can abstain from providing a label in addition to returning noisy labels. We provide a model for this setting where the abstention rate and the noise rate increase as we get closer to the decision boundary of the ground truth hypothesis. We provide an algorithm and an analysis of the number of queries it makes to the labeling oracle; finally we provide matching lower bounds to demonstrate that our algorithm has near-optimal estimation accuracy.