LPM-Net: Lightweight pixel-level modeling network based on CNN and Mamba for 3D medical image fusion

Abstract

Deep learning-based medical image fusion has become a prevalent approach to facilitate computer-aided diagnosis and treatment. The mainstream image fusion methods predominantly rely on encoder–decoder architectures and utilize unsupervised loss functions for training, resulting in the blurring or loss of fused image details and limited inference speed. To resolve these problems, this paper presents a pixel-level modeling network for effective fusion of 3D medical images. The network comprises three structurally identical branches: an unsupervised fusion branch and two supervised reconstruction branches. In the fusion branch, the feature extraction modules utilize the dense convolutional neural network and Mamba to extract image features based on axis decomposition. The base and detail components are then predicted from these extracted features and fused to generate the fused image pixel by pixel. Notably, two reconstruction branches share the parameters of feature extraction modules with the fusion branch and provide the supervised loss, which is integrated with the unsupervised loss to enhance the fusion performance. The experiments on six datasets of multiple modalities and organs demonstrates that our method achieves effective medical image fusion by preserving image details effectively, minimizing image blurring and reducing the number of parameters. Meanwhile, our method has significant advantages in eight fusion metrics over the compared mainstream methods, and it provides relatively fast inference speed (e.g., 90 volumes/s on the BraTS2020 dataset). Indeed, our method will provide valuable means to improve the accuracy and efficiency of image fusion-based diagnosis and treatment systems. The source code is available on GitHub at https://github.com/coolllcat/LPM-Net.