Rural Medical Centers Struggle to Produce Well-Calibrated Clinical Prediction Models: Data Augmentation Can Help.

Abstract

Machine learning models support many clinical tasks; however, challenges arise with the transportability of these models across a network of healthcare sites. While there are guidelines for updating models to account for local context, we hypothesize that not all healthcare organizations, especially those in smaller and rural communities, have the necessary patient volumes to facilitate local fine tuning to ensure models are reliable for their populations. To investigate these challenges, we conducted an experiment using data from a real network of hospitals to predict 30-day unplanned hospital readmission and a simulation study using data from a multi-site ICU dataset to evaluate the utility of synthetic data generation (SDG) to augment local data volumes. Several factors associated with rurality were correlated with model miscalibration and rural sites failed to meet sample size requirements for local recalibration. Our results indicate that deep learning approaches to SDG produced the best local classifiers.