Comparison of machine learning and human prediction to identify trauma patients in need of hemorrhage control resuscitation (ShockMatrix study): a prospective observational study

Background

Machine learning could improve the timely identification of trauma patients in need of hemorrhage control resuscitation (HCR), but the real-life performance remains unknown. The ShockMatrix study aimed to compare the predictive performance of a machine learning algorithm with that of clinicians in identifying the need for HCR.

Methods

Prospective, observational study in eight level-1 trauma centers. Upon receiving a prealert call, trauma clinicians in the resuscitation room entered nine predictor variables into a dedicated smartphone app and provided a subjective prediction of the need for HCR. These predictors matched those used in the machine learning model. The primary outcome, need for HCR, was defined as: transfusion in the resuscitation room, transfusion of more than four red blood cell units in 6 h of admission, any hemorrhage control procedure within 6 h, or death from hemorrhage within 24 h. The human and machine learning performances were assessed by sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, and net clinical benefit. Human and machine learning agreement was assessed with Cohen's kappa coefficient.

Findings

Between August 2022 and June 2024, out of 5550 potential eligible patients, 1292 were ultimately included in the analyses. The need for HCR occurred in 170/1292 patients (13%). The results showed a positive likelihood ratio of 3.74 (95% confidence interval [CI]: 3.20–4.36) and a negative likelihood ratio of 0.36 (95% CI: 0.29–0.46) for the human prediction and a positive likelihood ratio of 4.01 (95% CI: 3.43–4.70) and negative likelihood ratio of 0.35 (95% CI: 0.38–0.44) for the machine learning prediction. The combined use of human and machine learning prediction yielded a sensitivity of 83% (95% CI: 77–88%) and a specificity of 73% (95% CI: 70–75%). The Cohen's kappa coefficient showed an agreement of 0.51 (95% CI: 0.48–0.55).

Interpretation

The prospective ShockMatrix temporal validation study suggests a comparable human and machine learning performance to predict the need for HCR using real-life and real-time information with a moderate level of agreement between the two. Machine learning enhanced decision awareness could potentially improve the detection of patients in need of HCR if used by clinicians.

Funding

The study received no funding.