Temporal distribution shift in real-world pharmaceutical data: Implications for uncertainty quantification in QSAR models

The estimation of uncertainties associated with predictions from quantitative structure–activity relationship (QSAR) models can accelerate the drug discovery process by identifying promising experiments and allowing an efficient allocation of resources. Several computational tools exist that estimate the predictive uncertainty in machine learning models. However, deviations from the i.i.d. setting have been shown to impair the performance of these uncertainty quantification methods. We use a real-world pharmaceutical dataset to address the pressing need for a comprehensive, large-scale evaluation of uncertainty quantification approaches in the context of realistic distribution shifts over time. We investigate the performance of several popular uncertainty estimation methods for classification models, including ensemble-based and Bayesian approaches. Furthermore, we use this real-world setting to systematically assess the distribution shifts in label and descriptor space and their impact on the capability of the uncertainty quantification methods. Our study reveals significant shifts over time in both label and descriptor space and a clear connection between the magnitude of the shift and the nature of the assay. Moreover, we show that pronounced distribution shifts impair the performance of popular uncertainty quantification methods used in QSAR models. This work highlights the challenges of identifying uncertainty quantification techniques that remain reliable under distribution shifts introduced by real-world data.