Calibration in machine learning uncertainty quantification: Beyond consistency to target adaptivity

Abstract

Reliable uncertainty quantification (UQ) in machine learning (ML) regression tasks is becoming the focus of many studies in materials and chemical science. It is now well understood that average calibration is insufficient, and most studies implement additional methods for testing the conditional calibration with respect to uncertainty, i.e., consistency. Consistency is assessed mostly by so-called reliability diagrams. There exists, however, another way beyond average calibration, which is conditional calibration with respect to input features, i.e., adaptivity. In practice, adaptivity is the main concern of the final users of the ML-UQ method, seeking the reliability of predictions and uncertainties for any point in the feature space. This article aims to show that consistency and adaptivity are complementary validation targets and that good consistency does not imply good adaptivity. An integrated validation framework is proposed and illustrated with a representative example.