Denoising diffusion based infrared and visible image fusion with transformer

Abstract

Infrared-visible image fusion (IVF) aims to combine the complementary information from infrared and visible images. In the field of image fusion, generative adversarial networks (GANs) have achieved promising results. However, the issues of unstable training and mode collapse remain challenging to resolve. Therefore, we propose a novel IVF method based on a diffusion model combined with fusion knowledge priors, termed DDFT. DDFT is divided into two parts, a pre-fusion module and a diffusion model. Specifically, it first obtains images containing the prior distribution of the fusion task through a pre-fusion module. Subsequently, the forward diffusion process gradually removes distinguishable features from the output of the pre-fusion module, The reverse diffusion process learns the fusion knowledge prior distribution and leverages a Transformer module to capture global features, generating high-quality fused images. Comparative experiments demonstrate that DDFT excels in IVF tasks, especially in preserving weak textures. Generalization experiments illustrate that DDFT preserves image features in both simple and complex environments. Ablation experiments further validated the crucial role of the fusion prior information obtained through pre-fusion in DDFT. Compared to existing diffusion models that are only used as feature extractors or with fixed parameters, DDFT is the first to achieve an end-to-end trainable diffusion framework for directly generating fused images.