Using machine learning and natural language processing in triage for prediction of clinical disposition in the emergency department.

Abstract

Background: Accurate triage is required for efficient allocation of resources and to decrease patients' length of stay. Triage decisions are often subjective and vary by provider, leading to patients being over-triaged or under-triaged. This study developed machine learning models that incorporated natural language processing (NLP) to predict patient disposition. The models were assessed by comparing their performance with the judgements of emergency physicians (EPs).

Method: This retrospective study obtained data from patients visiting EDs between January 2018 and December 2019. Internal validation data came from China Medical University Hospital (CMUH), while external validation data were obtained from Asia University Hospital (AUH). Nontrauma patients aged ≥ 20 years were included. The models were trained using structured data and unstructured data (free-text notes) processed by NLP. The primary outcome was death in the ED or admission to the intensive care unit, and the secondary outcome was either admission to a general ward or transferal to another hospital. Six machine learning models (CatBoost, Light Gradient Boosting Machine, Logistic Regression, Random Forest, Extremely Randomized Trees, and Gradient Boosting) and one Logistic Regression derived from triage level were developed and evaluated using EPs' predictions as reference.

Result: A total of 17,2101 and 41,883 patients were enrolled from CMUH and AUH, respectively. EPs achieved F1 core of 0.361 and 0.498 for the primary and secondary outcomes, respectively. All machine learning models achieved higher F1 scores compared to EPs and Logistic Regression derived from triage level. Random Forest was selected for further evaluation and fine-tuning, because of its robust calibration and predictive performance. In internal validation, it achieved Brier scores of 0.072 and 0.089 for the primary and secondary outcomes, respectively, and 0.076 and 0.095 in external validation. Further analysis revealed that incorporating unstructured data significantly enhanced the model's performance. Threshold adjustments were applied to improve clinical applicability, aiming to balance the trade-off between sensitivity and positive predictive value.

Conclusion: This study developed and validated machine learning models that integrate structured and unstructured triage data to predict patient dispositions, distinguishing between general ward and critical conditions like ICU admissions and ED deaths. Integrating both structured and unstructured data significantly improved model performance.