Derivation and Experimental Performance of Standard and Novel Uncertainty Calibration Techniques.

Abstract

To aid in the transparency of state-of-the-art machine learning models, there has been considerable research performed in uncertainty quantification (UQ). UQ aims to quantify what a model does not know by measuring variation of the model under stochastic conditions and has been demonstrated to be a potentially powerful tool for medical AI. Evaluation of UQ, however, is largely constrained to visual analysis. In this work, we expand upon the Rejection Classification Index (RC-Index) and introduce the relative RC-Index as measures of uncertainty based on rejection classification curves. We hypothesize that rejection classification curves can be used as a basis to derive a metric of how well a given arbitrary uncertainty quantification metric can identify potentially incorrect predictions by an ML model. We compare RC-Index and rRC-Index to established measures based on lift curves.