UIRGBfuse: Revisiting infrared and visible image fusion from the unified fusion of infrared channel with R, G, and B channels

Abstract

Infrared and visible image fusion aims to obtain fused images with complementary information from infrared and visible modalities. The visible image captured by the visible spectrum camera consists of R, G, and B channels, exhibiting color information. However, existing fusion frameworks for infrared and visible images typically treat the fusion task as the fusion of infrared images with single-channel grayscale visible images. This approach neglects the fact that different gradient distributions between R, G, and B channels of RGB visible images, which can result in unnatural fusion effects, distortion, poor preservation of details from source images, and degradation of color fidelity. To achieve superior fusion performance in infrared and RGB visible image fusion, a unified fusion framework called UIRGBfuse is proposed in this study. It fused the infrared image with the R, G, and B channels through a unified fusion approach, along with an IR-RGB joint fusion learning strategy that has been designed to ensure natural and outstanding fusion results. The UIRGBfuse consists of separate branches for feature extraction and feature fusion, creating a cohesive architecture for fusing the infrared channel with the R, G, and B channels. Additionally, the training process is guided by R, G, and B fusion losses as part of the devised IR-RGB joint fusion learning strategy. In addition, this study implements the frequency domain compensate feature fusion module to achieve desirable feature fusion performance by the compensate features obtained from the frequency domain. Furthermore, the hybrid CNN-Transformer deep feature refinement module is realized in this study to refine the deep fused features obtained from the fusion branches, thereby further enhancing the fusion performance of UIRGBfuse. Moreover, to address color fidelity distortion observed in infrared and RGB visible image fusion, an adaptive cross-feature fusion reconstructor with the capability of adaptively fusing multi-branch fusion features is constructed in this work. Ablation studies have been conducted on publicly available datasets to validate the effectiveness of the proposed unified fusion architecture, IR-RGB joint fusion learning strategy, feature fusion and refinement modules, and reconstructor. The superiority of the proposed UIRGBfuse over other representative state-of-the-art infrared and visible image fusion methods in terms of natural fusion, retention of source image details, and color fidelity has been demonstrated through comparison and generalization experiments. Finally, object detection experiments have shown that the fused images obtained by UIRGBfuse are capable of successfully detecting more targets than other competitors.