Real-world multimetric comparison of four commercial artificial intelligence solutions for intracranial hemorrhage detection.

Abstract

Purpose: To evaluate the real-world multimetric performance of four commercially available computed tomography (CT)-based artificial intelligence (AI) solutions for acute intracranial hemorrhage (AIH).

Methods: Patients who underwent non-contrast brain CT for suspected AIH in our emergency room between February and March 2024 were screened. After applying the inclusion and exclusion criteria, 436 CT scans were included in the final analysis. Three neuroradiologists established the ground truth for AIH and hemorrhage volume. For detection performance, the area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (AUPRC), and Brier score were calculated based on the available probability score, whereas sensitivity, specificity, precision, and F1 score were calculated based on binary classification. Bland-Altman analysis was performed to assess volumetric agreement for AIH between each algorithm's calculations and the neuroradiologists' measurements.

Results: A total of 436 patients (mean age, 62 years ± 20; male patients, 209) were enrolled. The AUROC (0.96 to 0.99) and sensitivity (0.85 to 0.92) were high across all solutions, with no statistically significant differences in pairwise comparisons (P > 0.05). However, solution B demonstrated the highest AUPRC [0.98, 95% confidence interval (CI): 0.94, 1.00] and the lowest Brier score [0.02 (95% CI: 0.02, 0.03)]. In binary performance, both solutions B and D exhibited significantly higher specificity (1.00 and 0.99), precision (0.90 to 0.98), and F1 score (0.87 to 0.94) than the other solutions (P < 0.05). For volumetric agreement of AIH, solution D showed the lowest mean difference [-0.87 mm³ (95% CI: -1.47, -0.27)] and the narrowest limits of agreement (-13.4 to 11.6) relative to the neuroradiologists' measurements.

Conclusion: In a real-world emergency setting, all four commercially available CT-based AI solutions for AIH demonstrated uniformly excellent performance; however, meaningful differences emerged in confirmatory performance and volumetric agreement. These distinct, algorithm-specific trade-offs provide practical guidance for selecting and integrating appropriate AI solutions to improve AIH diagnosis and management workflows.

Clinical significance: The algorithm-specific performance trade-offs identified in this study suggest that no single AI solution is universally optimal; solutions with superior confirmatory performance may reduce unnecessary notifications in high-volume emergency settings, whereas those with more consistent volumetric agreement may better support treatment planning and longitudinal monitoring. A structured, multimetric evaluation aligned with institutional priorities is essential for evidence-based AI procurement in acute stroke imaging.