Evaluation of AI diagnostic systems for breast ultrasound: comparative analysis with radiologists and the effect of AI assistance.

Abstract

Purpose: The purpose of this study is to evaluate the diagnostic accuracy of an artificial intelligence (AI)-based Computer-Aided Diagnosis (CADx) system for breast ultrasound, compare its performance with radiologists, and assess the effect of AI-assisted diagnosis. This study aims to investigate the system's ability to differentiate between benign and malignant breast masses among Japanese patients.

Materials and methods: This retrospective study included 171 breast mass ultrasound images (92 benign, 79 malignant). The AI system, BU-CAD™, provided Breast Imaging Reporting and Data System (BI-RADS) categorization, which was compared with the performance of three radiologists. Diagnostic accuracy, sensitivity, specificity, and area under the curve (AUC) were analyzed. Radiologists' diagnostic performance with and without AI assistance was also compared, and their reading time was measured using a stopwatch.

Results: The AI system demonstrated a sensitivity of 91.1%, specificity of 92.4%, and an AUC of 0.948. It showed comparable diagnostic performance to Radiologist 1, with 10 years of experience in breast imaging (0.948 vs. 0.950; p = 0.893), and superior performance to Radiologist 2 (7 years of experience, 0.948 vs. 0.881; p = 0.015) and Radiologist 3 (3 years of experience, 0.948 vs. 0.832; p = 0.001). When comparing diagnostic performance with and without AI, the use of AI significantly improved the AUC for Radiologists 2 and 3 (p = 0.001 and 0.005, respectively). However, there was no significant difference for Radiologist 1 (p = 0.139). In terms of diagnosis time, the use of AI reduced the reading time for all radiologists. Although there was no significant difference in diagnostic performance between AI and Radiologist 1, the use of AI substantially decreased the diagnosis time for Radiologist 1 as well.

Conclusion: The AI system significantly improved diagnostic efficiency and accuracy, particularly for junior radiologists, highlighting its potential clinical utility in breast ultrasound diagnostics.