MMBCFNet: multi modal hybrid deep learning framework for breast cancer detection using MRI, mammography, and ultrasound images.

Abstract

Accurate diagnosis of breast cancer is essential for enhancing the outcomes of patients. Although magnetic resonance imaging (MRI), mammography (MMG), and ultrasound (US) provides complementary diagnostic data, current deep learning technologies tend to use only one of these methods or do not have an efficient means of fusing diagnostic data, which restricts the diagnostic capabilities. This study proposes Multimodal Breast Cancer Fusion Network (MMBCFNet), a novel multi modal deep learning that combines MRI, MMG, and US with the help of hybrid models that are modality-specific and an attention-based feature fusion approach. In particular, MRI features are obtained with the help of DSMRINet, a combination of 3D DenseNet and Swin Transformer models; MMG features are obtained with the help of ERMMGNet, which is a combination of ResNet and EfficientNet models; and US features are obtained with the help of MRUSNet, which is a combination of MobileNetV3 and ResNeXt models. The predictions obtained from these modality-specific networks are combined using a weighted decision-level fusion mechanism to generate the final diagnostic outcome. Experimental results demonstrate that the proposed framework achieves accuracy of 98.78%, precision of 98.29%, sensitivity of 98.54%, specificity of 98.46%, and MCC of 0.9868, outperforming baseline models. The evaluation results demonstrate that the proposed framework effectively leverages complementary information from multiple imaging modalities and determines improved strength and accuracy for breast cancer detection.