Investigating Normalized Conformal Regressors

Abstract

Conformal prediction can be applied on top of any machine learning predictive regression model, thus turning it into a conformal regressor. Given a significance level $\epsilon$, conformal regressors output valid prediction intervals, i.e., the probability that the interval covers the true value is exactly $1-\epsilon$. To obtain validity, a calibration set that is not used for training the model must be set aside. In standard inductive conformal regression, the size of the prediction intervals is then determined by the absolute error made by the predictive model on a specific instance in the calibration set, where different significance levels correspond to different instances. In this setting, all prediction intervals will have the same size, making the resulting models very unspecific. When adding a technique called normalization, however, the difficulty of each instance is estimated, and the interval sizes are adjusted accordingly. An integral part of normalized conformal regressors is a parameter called $\beta$, which determines the relative importance of the difficulty estimation and the error of the model. In this study, the effects of different underlying models, difficulty estimation functions and $\beta$ -values are investigated. The results from a large empirical study, using twenty publicly available data sets, show that better difficulty estimation functions will lead to both tighter and more specific prediction intervals. Furthermore, it is found that the $\beta$ -values used strongly affect the conformal regressor. While there is no specific $\beta$ -value that will always minimize the interval sizes, lower $\beta$ -values lead to more variation in the interval sizes, i.e., more specific models. In addition, the analysis also identifies that the normalization procedure introduces a small but unfortunate bias in the models. More specifically, normalization using low $\beta$ -values means that smaller intervals are more likely to be erroneous, while the opposite is true for higher $\beta$ -values.