Multi-task learning for multi-scale breast cancer ultrasound image segmentation and classification based on visual perception

Breast tumor segmentation and classification are essential components of breast ultrasound (BUS) computer-aided diagnosis (CAD) systems, which help improve the accuracy of breast cancer diagnoses. However, challenges arise due to the complexity of tumor features, the intensity similarities between lesions and surrounding tissues, as well as variations in tumor shape and location. While deep learning has been widely applied in CAD systems, many existing approaches overlook the relationship between segmentation and classification tasks, limiting their effectiveness. We propose PMSMT-Net, a Perception-based Multi-scale Ultrasound Image Segmentation and Classification Multi-task Learning Network, which enhances both segmentation and classification performance in BUS images. The segmentation network integrates a Visual Perception Module (VPM) to simulate human-like focus on regions of interest and, combined with Multi-scale Dilated Convolution (MSDC), accurately captures morphological, locational, and edge features. To further improve segmentation accuracy, we introduce the Variable Residual Convolutional Block Attention Module (VR-CBAM) and the Receptive Field Block-based Perceptually Separable Convolution Module (RFB-PSC), which enhance context feature fusion and reduce spatial information loss. The VPM output, along with the segmentation results, is then used as input to the classification network, where a transfer learning and ensemble learning approach classifies breast tumors. Compared to state-of-the-art methods, PMSMT-Net achieves an average improvement of 3.7% in Dice coefficient and 3.6% in classification accuracy on two public BUS datasets. These results demonstrate the proposed model can significantly advance BUS-based tumor analysis and is of great significance for improving diagnostic precision and patient outcomes.