Training Confidence-Calibrated Classifier via Distributionally Robust Learning

Supervised learning via empirical risk minimization, despite its solid theoretical foundations, faces a major challenge in generalization capability, which limits its application in real-world data science problems. In particular, current models fail to distinguish in-distribution and out-of-distribution and give over confident predictions for out-of-distribution samples. In this paper, we propose an distributionally robust learning method to train classifiers via solving an unconstrained minimax game between an adversary test distribution and a hypothesis. We showed the theoretical generalization performance guarantees, and empirically, our learned classifier when coupled with thresholded detectors, can efficiently detect out-of-distribution samples.