Evaluation of a Multi-Instant Multimodal Artificial Intelligence System Supporting Interpretive and Noninterpretive Functions.

Abstract

Objective: Artificial intelligence (AI) has been shown to hold promise for improving breast cancer screening, offering advanced capabilities to enhance diagnostic accuracy and efficiency. This study aimed to evaluate the impact of a multimodal multi-instant AI-based system on the diagnostic performance of radiologists in interpreting mammograms.

Methods: We designed a multireader multicase study taking into account the evaluation of both interpretive and noninterpretive tasks. The study was approved by an institutional review board and is compliant with HIPAA. The dataset included 90 cancer-proven and 150 negative cases. The overall diagnostic performance was compared between the unaided vs aided reading condition. Intraclass correlation coefficient (ICC), Fleiss's kappa, and accuracy were used to quantify the agreement and performance on noninterpretive tasks. Reading time and perceived fatigue were used as comprehensive metrics to assess the efficiency of readers.

Results: The average area under the receiver operating characteristic curve increased by 7.4% (95% CI, 4.5%-10%) with the concurrent assistance of the AI system (P <.001). On average, readers found 8% more cancers in the assisted reading condition. The ICC went from 0.6 (95% CI, 0.55-0.65) in the unassisted condition to 0.74 (95% CI, 0.70-0.78) for readings done with AI (P <.001). An overall decrease of 24% in reading time and a reduction in perceived fatigue was also found.

Conclusion: The incorporation of this AI system, capable of handling multiple image type, prior mammograms, and multiple outputs, improved the diagnostic proficiency of radiologists in identifying breast cancer while also reducing the time required for combined interpretive and noninterpretive tasks.