Rethinking the approach to lightweight multi-branch heterogeneous image fusion frameworks: Infrared and visible image fusion via the parallel Mamba-KAN framework

The infrared and visible image fusion (IVIF) technique, as an important branch of image processing, has garnered significant attention due to its ability to capture thermal radiation features in low-light conditions and combine them with the rich detail and color information of visible images. However, challenges remain in this field, including the difficulty of balancing information from both modalities using homogenous fusion strategies and the increasing model complexity, which affects computational efficiency. To address these issues, we present an innovative lightweight IVIF framework based on the parallel Mamba-KAN (PMKFuse) model, featuring a multi-branch heterogeneous model design. By integrating our proposed multi-channel parallel cross-vision Mamba (PCVM) modules with parallel KAGtention (PKAGN) modules, our approach effectively extracts features at both global and local levels. This not only ensures high performance in image fusion tasks but also significantly reduces the number of model parameters. Additionally, a composite loss function is developed, integrating intensity, gradient, and feature decomposition losses to optimize the training process. The experimental results demonstrate that PMKFuse not only minimizes the number of model parameters but also outperforms the current state-of-the-art (SOTA) methods in both subjective visual evaluation and objective performance metrics for IVIF. These findings highlight the model’s effectiveness in improving fusion quality and its potential for wide-ranging practical applications, advancing the field of image processing. The codes are available at https://github.com/sunyichen1994/PMKFuse.