Conceptual framework for prediction models of patient deterioration based on nursing documentation patterns: reproducibility and generalizability with a large number of hospitals across the United States

Abstract

Objective

The Health Process Model (HPM)-ExpertSignals Conceptual Framework posits that healthcare professionals’ patient care behaviors can be used to predict in-hospital deterioration. Prediction models based on this framework have been validated using data from 4 hospitals within two healthcare systems. As clinician-system interactions may differ across organizations, this study aimed to evaluate the reproducibility and generalizability of the underlying conceptual framework using data from over 200 hospitals across the US.

Methods

This study used eICU-CRD, a publicly accessible dataset with data from 208 US hospitals. A logistic regression model was developed to predict in-hospital deterioration following the HPM-ExpertSignals conceptual framework. To test its reproducibility, patients were randomly split into training and testing datasets. After bootstrap testing of the model, the mean precision-recall curve (AUPRC) was compared with outcomes from previously published studies. For generalizability testing, the hospitals in the dataset were randomly assigned into model training or testing sets. After the model was trained with training hospitals’ data, generalizability was measured as the percentage of testing hospitals with an AUPRC at or above a baseline performance obtained in the reproducibility experiment.

Results

The AUPRC in the reproducibility experiment was 0.10 (0.09,0.11; 95% CI), equivalent to the AUPRC reported in a previous study at 0.093 (0.09, 0.096; 95% CI). In the generalizability experiment, 94% of the testing hospitals had AUPRC at or above the baseline AUPRC of 0.10.

Conclusion

The study provides evidence supporting the reproducibility of a predictive model following the HPM-ExpertSignals framework. This model also generalized to most hospitals without additional training. Nevertheless, some hospitals still obtained lower-than-expected performance, highlighting the need for model evaluation and potential fine-tuning before local adoption. Similar studies are needed to investigate the reproducibility and generalizability of other classes of machine learning models in healthcare.