Multimodal Medical Image Fusion With UNet-Based Multi-Scale Transformer Networks

Abstract

Multimodal medical image fusion can generate medical images that contain both functional metabolic information and structural tissue details, thereby providing doctors with more comprehensive information. Current deep learning-based methods often employ convolutional neural networks (CNNs) for feature extraction. However, CNNs exhibit limitations in capturing global contextual information compared to Transformers. Moreover, single-scale networks fail to exploit the complementary information between different scales, which limits their ability to fully capture rich image features and results in suboptimal fusion performance. To address these limitations, this paper proposes a multimodal medical image fusion method with UNet-based multi-scale Transformer network. First, we design a UNet-based encoder that incorporates a lightweight Transformer model, PVTv2, to extract multi-scale features from both MRI and SPECT images. To enhance the structural details of MRI images, we introduce the Edge-Guided Attention Module. Additionally, we propose an objective function that combines structural and pixel-level losses to optimize the proposed network. We perform both qualitative and quantitative experiments on mainstream datasets, and the results demonstrate that the proposed method outperforms several representative methods. In addition, we extend the proposed method to other biomedical functional and structural image fusion tasks, and the results show that the proposed method has good generalization capability.