A comparative performance analysis of live clinical triage using rules-based triage protocols versus artificial intelligence-based automated virtual triage

Abstract

Objective: Compare the triage care referral accuracy of artificial intelligence (AI) based virtual triage (VT) to rules-based triage protocols (RBTP) live telephonic triage.Methods: Clinical vignettes were selected for a comparison of care referral accuracy of RBTPs with a widely utilized AI-based VT solution. Vignettes (149) included patient complaints, expected triage and urgency assessment. Triage levels were mapped to three triage categories (urgent care, non-emergent care and self-care). Each vignette was evaluated/completed using AI-based VT and RBTP triage modalities by a total of four physicians in series, with independent assessment for errors and inconsistencies. Triage assessment precision was analyzed by matching the expected triage assessment, sensitivity and F1 scores (harmonic mean of precision and recall).Results: Both modalities achieved > 70% triage accuracy, and safety performance was identical at 91%. AI-based VT was more accurate in care referral for emergency and non-emergency care and overtriaged to emergency care 50% less frequently than RBTP, but was less accurate than RBTP in self-care vignettes (neither statistically significant). Both modalities demonstrated decreased sensitivity as care urgency/acuity decreased, more pronounced in AI-based VT than RBTP. AI-based VT captured four times as much information and data as RBTP.Conclusions: AI-based VT and RBTP were comparable in care referral accuracy and disposition safety. While AI-based VT provides accurate and safe triage recommendations at a lower total cost, care organizations should assess how AI-based VT compares to a live clinical triage capability with respect to organizational priorities, budgetary considerations, characteristics of the patient/member population served, and the existing technological environment.