Development and internal validation of triage-stage prediction models for short-term critical outcomes among adult emergency department patients entering observation or hospital-based care: a retrospective study.

Abstract

Background: Risk stratification at emergency department triage may help identify patients who require closer assessment or monitoring. However, prediction models that rely only on information available at the triage stage remain less well evaluated. This study aimed to develop and internally validate machine learning models based on triage-stage information for predicting short-term critical outcomes in adult emergency department patients and to interpret a representative explainable model using SHAP analysis.

Methods: This retrospective observational study included 1,842 consecutive adult patients who completed standardized electronic triage and subsequently entered an emergency observation or hospital-based care pathway at a tertiary general hospital between January 2022 and December 2025. The prediction time point and landmark were defined as completion of electronic emergency triage. Only patients who were free of all outcome components at the landmark were included, and candidate predictors were restricted to routinely available information recorded no later than triage completion. The primary outcome was the first critical event occurring during the subsequent 72 h: intensive care unit admission, endotracheal intubation or invasive mechanical ventilation, cardiopulmonary resuscitation, objectively documented escalation involving an implemented resuscitation-level intervention, or in-hospital death. Additional sensitivity analyses restricted the outcome to intensive care unit admission, intubation or invasive ventilation, cardiopulmonary resuscitation, or death and refitted the model without emergency triage level. The cohort was randomly divided into a development cohort and a 30% hold-out cohort. Because the hold-out cohort had already been examined during the original analysis, it was not considered an independent validation set. The primary internal-validation analysis therefore consisted of 10 repetitions of nested cross-validation within the development cohort, using five stratified outer folds for performance estimation and four stratified inner folds for preprocessing and hyperparameter tuning. Logistic regression, random forest, support vector machine, XGBoost, and LightGBM were evaluated using the same resampling framework. The previously examined exploratory hold-out cohort was retained only for secondary exploratory analysis. Model performance was assessed using discrimination, calibration, classification metrics, Brier score, decision-curve analysis, and SHAP-based interpretation. Bootstrap optimism correction was additionally used to assess overfitting.

Results: All 1,842 included patients were event-free at triage completion, and 214 experienced a short-term critical outcome during the subsequent 72-hour period, corresponding to an event rate of 11.6%. The development cohort included 1,289 patients with 150 events, and the previously examined exploratory hold-out cohort included 553 patients with 64 events. In 10 repetitions of nested cross-validation, the mean AUCs were 0.821 ± 0.012 for logistic regression, 0.817 ± 0.015 for random forest, 0.752 ± 0.018 for support vector machine, 0.831 ± 0.013 for XGBoost, and 0.833 ± 0.014 for LightGBM. XGBoost was highest ranked in 42% of repeated outer-validation results, LightGBM in 48%, and logistic regression in 10%. These primary internal-validation results supported broadly comparable performance among XGBoost, LightGBM, and logistic regression rather than consistent superiority of one algorithm. In the secondary exploratory analysis, the representative XGBoost model achieved an AUC of 0.828 in the previously examined exploratory hold-out cohort. At the development-derived threshold of 0.106, its accuracy was 0.770, sensitivity 0.656, specificity 0.785, positive predictive value 0.286, negative predictive value 0.946, F1 score 0.398, and Brier score 0.080. Its paired comparisons, sensitivity analyses, calibration, decision-curve analysis, and SHAP findings in this cohort were interpreted as exploratory.

Conclusions: Repeated nested cross-validation showed broadly comparable discrimination for XGBoost, LightGBM, and logistic regression in predicting post-triage short-term critical outcomes. XGBoost was retained as a representative explainable model, whereas its performance and benchmark comparisons in the previously examined exploratory hold-out cohort should be regarded as secondary exploratory analyses. The study does not establish the superiority or independent validation of XGBoost. Evaluation in a genuinely unused temporal cohort and external prospective validation remain necessary before clinical implementation.