Comprehensive performance testing and external validation of an AI algorithm to detect and segment brain metastases.

Abstract

Background: Artificial intelligence (AI)-based models have shown initial promise in imaging brain metastasis; however, many lack validation against advanced imaging-informed datasets, precluding external validity and limiting widespread adoption. To overcome these limitations, we performed comprehensive performance testing against reference standard metrics and externally validated an AI algorithm.

Methods: As part of its FDA-clearance process, performance testing of a previously developed U-Net-based AI model was conducted on a multi-institutional cohort with reference standard established via consensus review by 3 neuroradiologists. External validation was performed on patients imaged with dual sequences (augmented) as well as an open-access dataset (UCSF-BMSR). Evaluation metrics included sensitivity, false positive (FP) rate, positive predictive value (PPV), dice similarity coefficient (DSC), 95% Hausdorff distance, normalized surface distance, and qualitative physician assessment.

Results: In the FDA performance testing cohort, the AI algorithm achieved a sensitivity of 90.0% (95% CI: 87.0%-94.0%), DSC of 0.86 (95% CI: 0.83-0.89), and average FP rate of 0.57 lesions. In the augmented and open-access external validation cohorts, a sensitivity of 81.4% (95% CI: 73.7%-89.1%) and 85.2% (95% CI: 83.0%-87.4%) with an average number of 0.22 and 1.19 FP lesions and DSCs of 0.70 (95% CI: 0.66-0.73) and 0.78 (95% CI: 0.77-0.78) were calculated, respectively. In the augmented external validation cohort, 38.5% of contours were rated as requiring major revisions.

Conclusion: This AI algorithm demonstrated promising performance via 3 unique datasets. However, given the notable rate of contour revisions, these findings support its clinical role not as an autonomous system but as a human-in-the-loop tool requiring physician oversight.