SSEFusion: Salient semantic enhancement for multimodal medical image fusion with Mamba and dynamic spiking neural networks

Abstract

Multimodal medical image fusion technology enhances medical representations and plays a vital role in clinical diagnosis. However, fusing medical images remains a challenge due to the stochastic nature of lesions and the complex structures of organs. Although many fusion methods have been proposed recently, most struggle to effectively establish global context dependency while preserving salient semantic features, leading to the loss of crucial medical information. Therefore, we propose a novel salient semantic enhancement fusion (SSEFusion) framework, whose key components include a dual-branch encoder that combines Mamba and spiking neural network (SNN) models (Mamba-SNN encoder), feature interaction attention (FIA) blocks, and a decoder equipped with detail enhancement (DE) blocks. In the encoder, the Mamba-based branch introduces visual state space (VSS) blocks to efficiently establish global dependencies and extract global features for the effective identification of the lesion area. Meanwhile, the SNN-based branch dynamically extracts fine-grained salient features to enhance the retention of medical semantic information in complex structures. Global features and fine-grained salient features semantically interact to achieve feature complementarity through the FIA blocks. Benefiting from the DE block, SSEFusion generates fused images with prominent edge textures. Furthermore, we propose a salient semantic loss based on leaky-integrate-and-fire (LIF) neurons to enhance the guidance in extracting key features. Extensive fusion experiments show that SSEFusion outperforms state-of-the-art fusion methods in terms of salient medical semantic information retention. The code is available at https://github.com/Shiqiang-Liu/SSEFusion.