Real world clinical experience of using Brainomix e-CTA software in a medium size acute NHS trust.

Abstract

Objectives: Artificial intelligence (AI) software including Brainomix "e-CTA" which detect large vessel occlusions (LVO) have clinical potential. We hypothesised that in real world use where prevalence is low, its clinical utility may be overstated.

Methods: In this single centre retrospective service evaluation project, data sent to Brainomix from a medium size acute National Health Service (NHS) Trust hospital between 1/3/2022-1/3/2023 was reviewed. 584 intracranial computed tomography angiogram (CTA) datasets were analysed for LVO by e-CTA. The e-CTA output and radiology report were compared to ground truth, defined by a consultant radiologist with fellowship neuroradiology training, with access to subsequent imaging and clinical notes. Sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) were calculated.

Results: Of 584 cases (45% female, mean age 70 ± 16 years), 9% (n = 50) had LVO. e-CTA had a sensitivity of 0.78 (95% CI 0.64-0.88), specificity of 0.93 (0.9-0.95), PPV of 0.5 (0.42-0.58) and NPV of 0.98 (0.96-0.99). e-CTA had an error rate of 9% (52/584). Erroneous cases were categorised into causes for error. Common causes for false positives included incorrect anatomy (21%, 8/39) and other pathology (13%, 5/39), with several uncategorisable cases (39%, 15/39). Common causes for false negatives included LVO within the terminal internal carotid artery (ICA) (55%, 6/11) and uncategorisable (18%, 2/11).

Conclusions: We demonstrated that PPV of e-CTA is poor in consecutive cases in a real-world NHS setting. We advocate for local validation of AI software prior to clinical use.

Advances in knowledge: Common AI errors were due to anatomical misidentification, presence of other pathology, and misidentifying LVO in the terminal ICA.