A model-based infrared and visible image fusion network with cooperative optimization

Abstract

The primary objective of infrared and visible image fusion is to amalgamate information from multi-modal images into fused images containing salient targets and abundant details. Existing fusion methods predominantly either rely on a “black-box” model with the absence of interpretability and generalizability or focus on improving the visual appeal of images but ignore the semantic information. To address these shortcomings, this paper introduces a task-oriented infrared and visible image fusion network that integrates model-based and data-driven regularization with cooperative optimization. First, based on the unrolled cooperative optimization formulation, we design a cyclical cooperative training strategy to cascade the image fusion and semantic segmentation modules, thereby augmenting content information with semantic information. Then, considering both high computational efficiency of the deep neural network and interpretability of the traditional optimization model, we explicitly encode a well-studied fusion algorithm into an unrolling auto-encoder network with a dual private encoder for the fusion module. Additionally, our model is compact, providing an efficient solution under conditions of limited resources. A comprehensive qualitative and quantitative analysis of experimental results on image fusion and semantic-aware evaluation demonstrates that the proposed method completely preserves thermal targets and background texture details, surpassing state-of-the-art alternatives in terms of image quality and high-level semantics.