Comparing Machine Learning and Nurse Predictions for Hospital Admissions in a Multisite Emergency Care System

Abstract

Objective

To prospectively compare nurse predictions with a machine learning (ML) model for hospital admissions and evaluate whether adding the nurse prediction to ML outputs enhances predictive performance.

Patients and Methods

In this prospective, observational study at 6 hospitals in a large mixed quaternary/community emergency department (ED) system (annual ED census ∼500,000), triage nurses recorded a binary admission prediction for adult patients. These predictions were compared with an ensemble ML model (XGBoost + Bio-Clinical BERT) trained on structured data (demographics, vital signs, and medical history) and triage text. Nurse predictions were similarly analyzed and then integrated with the ML output to assess for improvement in predictive accuracy.

Results

The ensemble ML model (XGBoost + Bio-Clinical BERT) was trained on 1.8 million historical ED visits (January 2019 to December 2023). It was then tested on 46,912 prospective ED visits with recorded nurse predictions (September 1, 2024 to October 31, 2024). In the prospective arm, nurse predictions yielded an accuracy of 81.6% (95% CI, 81.3-81.9), a sensitivity of 64.8% (63.7-65.8), and a specificity of 85.7% (85.3-86.0). At a 0.30 probability threshold, the ML model attained an accuracy of 85.4% (85.0-85.7) and a sensitivity of 70.8% (69.8-71.7). Combining nurse predictions with the ML output did not improve accuracy beyond the model alone.

Conclusion

Machine learning-based predictions outperformed triage nurse estimates for hospital admissions. These findings suggest that an admission prediction system anchored by ML can perform reliably using data available at triage.