Rif-Diff: Improving image fusion based on diffusion model via residual prediction

Abstract

This paper proposes an image fusion framework Rif-Diff, which adopts several strategies and approaches to improve current fusion methods based on diffusion model. Rif-Diff employs residual images as the generation target of the diffusion model to optimize the model’s convergence process and enhance the fusion performance. For fusion tasks lacking ground truth, image fusion prior is utilized to facilitate the production of residual images. Simultaneously, to overcome the limitations of the model’s learning capacity imposed by training with image fusion prior, Rif-Diff introduces the idea of image restoration to enable the initial fused images to incorporate more expected information. Additionally, a dual-step decision module is designed to address the blurriness issue of fused images in existing multi-focus image fusion methods that do not rely on decision maps. Extensive experiments demonstrate the effectiveness of Rif-Diff across multiple fusion tasks including multi-focus image fusion, multi-exposure image fusion, and infrared-visible image fusion. The code is available at: https://github.com/peixuanWu/Rif-Diff.