MKDFusion: modality knowledge decoupled for infrared and visible image fusion

Abstract

The purpose of infrared and visible fusion is to integrate useful information from both infrared and visible images into a single image. The fused image should possess rich texture details and salient target information of the two images. Current image fusion algorithms primarily face two limitations: 1) The lack of decoupling between modality-agnostic and modality-specific knowledge during the feature extraction stage hinders the alignment of modality-agnostic knowledge and the differentiation of modality-specific knowledge. 2) The interaction between modality features is not sufficiently explored in the feature fusion stage, which inhibits the exploitation of complementary information. To address the above challenges, we propose a Modality Knowledge Decoupled (MKD) module in the feature extraction stage and a Cross-Modality Mamba Fusion (CMF) module in the feature fusion stage. In MKD, we first utilize a dual-branch network to extract modality-agnostic and modality-specific knowledge separately. Then, a pair of Knowledge Discriminators (KD) is constructed to minimize inter-modality irrelevant knowledge and maximize inter-modality relevant knowledge. In CMF, the interactions between different modality knowledge are learnt in a hidden state space, which not only reduces the inter-modality knowledge differences but also enhances the texture information of the image. Experiments on three datasets demonstrate that our method outperforms existing methods, highlighting less salient targets and texture information more effectively. In addition, MKDFusion has demonstrated excellent generalization performance and enormous potential in high-level vision tasks in medical image fusion and object detection applications. The code is available at https://github.com/SEU-ZYC/MKDFusion.