A new integrated discrimination improvement index via odds

Abstract

Consider adding new covariates to an established binary regression model to improve prediction performance. Although difference in the area under the ROC curve (delta AUC) is typically used to evaluate the degree of improvement in such situations, its power is not high due to being a rank-based statistic. As an alternative to delta AUC, integrated discrimination improvement (IDI) has been proposed by Pencina et al. (2008). However, several papers have pointed out that IDI erroneously detects meaningless improvement. In the present study, we propose a novel index for prediction improvement having Fisher consistency, implying that it overcomes the problems in both delta AUC and IDI. Furthermore, our proposed index also has an advantage that the index we proposed in our previous study (Hayashi and Eguchi 2019) lacked: it does not require any hyperparameters or complicated transformations that would make interpretation difficult.