A multi-scale pyramid residual weight network for medical image fusion.

Abstract

Background: Due to the inherent limitations of imaging sensors, acquiring medical images that simultaneously provide functional metabolic information and detailed structural organization remains a significant challenge. Multi-modal image fusion has emerged as a critical technology for clinical diagnosis and surgical navigation, as it enables the integration of complementary information from different imaging modalities. However, existing deep learning (DL)-based fusion methods often face difficulties in effectively combining high-frequency detail information with low-frequency contextual information, which frequently leads to the degradation of high-frequency details. Therefore, there is a pressing need for a method that addresses these challenges, preserving both high- and low-frequency information while maintaining clear structural contours. In response to this issue, a novel convolutional neural network (CNN), named the multi-scale pyramid residual weight network (LYWNet), is proposed. The objective of this approach is to improve the fusion process by effectively integrating high- and low-frequency information, thereby enhancing the quality and accuracy of multimodal image fusion. This method aims to overcome the limitations of current fusion techniques and ensure the preservation of both functional and structural details, ultimately contributing to more precise clinical diagnoses and better surgical navigation outcomes.

Methods: We propose a novel CNN, LYWNet, designed to address these challenges. LYWNet is composed of three modules: (I) data preprocessing module: utilizes three convolutional layers to extract both deep and shallow features from the input images. (II) Feature extraction module: incorporates three identical multi-scale pyramid residual weight (LYW) blocks in series, each featuring three interactive branches to preserve high-frequency detail information effectively. (III) Image reconstruction module: utilizes a fusion algorithm based on feature distillation to ensure the effective integration of functional and anatomical information. The proposed image fusion algorithm enhances the interaction of contextual cues and retains the metabolic details from functional images while preserving texture details from anatomical images.

Results: The proposed LYWNet demonstrated its ability to retain high-frequency details during feature extraction, effectively combining them with low-frequency contextual information. The fusion results exhibited reduced differences between the fused image and the original images. The structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) were 0.5592±0.0536 and 17.3594±1.0211, respectively, for single-photon emission computed tomography-magnetic resonance imaging (SPECT-MRI), 0.5195±0.0730 and 14.5324±1.7365 for PET-MRI; 0.5376±0.0442 and 13.9202±0.7265 for magnetic resonance imaging-computed tomography.

Conclusions: LYWNet excels at integrating high-frequency detail information and low-frequency contextual information, addressing the deficiencies of existing DL-based image fusion methods. This approach provides superior fused images that retain the functional metabolic information and anatomical texture, making it a valuable tool for clinical diagnosis and surgical navigation.