F2Fusion: Frequency Feature Fusion Network for Infrared and Visible Image via Contourlet Transform and Mamba-UNet

Abstract

To integrate complementary thermal and texture information from source infrared (IR) and visible (VIS) images into a comprehensive fused image, traditional multiscale transform algorithms, and deep neural networks have been extensively explored for IR and VIS image fusion (IVIF). However, existing methods often face difficulties combining the strengths of these two approaches, particularly when it comes to balancing the preservation of salient and texture information in challenging conditions such as low light, glare, and overexposure. This article proposes a novel frequency feature fusion network (F2Fusion) that exploits detailed space-frequency transformation through contourlet transform (CT) and multiscale long-range learning via the Mamba-UNet architecture. The Mamba block is embedded into the multiscale encoder and decoder structures to improve feature extraction and image reconstruction performance. The CT operation replaces the conventional pooling layer in the multiscale encoder, converting spatial features into high- and low-frequency subbands. We then introduce a dual-branch frequency feature fusion module to facilitate the fusion of cross-modality illumination information and fine details based on the distinct characteristics of different frequency subbands. In addition, we design a composite loss function, which includes both gradient and salient constraints, to guide the precise synthesis of salient targets and texture regions. Qualitative and quantitative comparisons across three benchmark datasets demonstrate that the proposed method outperforms recent state-of-the-art (SOTA) fusion techniques. Extended experimental results on downstream object detection tasks further validate the distinct advantages of the proposed architecture for fusion through precise frequency decomposition. The code is available at: https://github.com/lrh-1994/F2Fusion