AFDFusion: An adaptive frequency decoupling fusion network for multi-modality image

Abstract

The multi-modality image fusion goal is to create a single image that provides a comprehensive scene description and conforms to visual perception by integrating complementary information about the merits of the different modalities, e.g., salient intensities of infrared images and detail textures of visible images. Although some works explore decoupled representations of multi-modality images, they struggle with complex nonlinear relationships, fine modal decoupling, and noise handling. To cope with this issue, we propose an adaptive frequency decoupling module to perceive the associative invariant and inherent specific among cross-modality by dynamically adjusting the learnable low frequency weight of the kernel. Specifically, we utilize a contrastive learning loss for restricting the solution space of feature decoupling to learn representations of both the invariant and specific in the multi-modality images. The underlying idea is that: in decoupling, low frequency features, which are similar in the representation space, should be pulled closer to each other, signifying the associative invariant, while high frequencies are pushed farther away, also indicating the intrinsic specific. Additionally, a multi-stage training manner is introduced into our framework to achieve decoupling and fusion. Stage I, MixEncoder and MixDecoder with the same architecture but different parameters are trained to perform decoupling and reconstruction supervised by the contrastive self-supervised mechanism. Stage II, two feature fusion modules are added to integrate the invariant and specific features and output the fused image. Extensive experiments demonstrated the proposed method superiority over the state-of-the-art methods in both qualitative and quantitative evaluation on two multi-modal image fusion tasks.