ILLVFusion: Infrared and low-light visible image fusion based on CNN and transformer

Abstract

In the field of image fusion, particularly in nocturnal or low-illumination conditions, the effective integration of infrared and visible light images to enhance image quality remains a pressing issue. This study introduces an innovative image fusion framework named ILLVFusion, which comprises three critical networks: the Low-Light Image Enhancement Network (LLIEN), the Fusion Network (FN), and the Brightness Feedback Network (BFN). The LLIEN module is designed to boost the brightness of low-light visible light images and reduce noise interference, thereby generating images with balanced color and lower noise levels. The fusion network FN employs a dual-branch encoder structure based on Convolutional Neural Network (CNN) and Transformer, aiming to efficiently extract global and detailed features from the input images. The BFN module provides brightness feedback to optimize the output of the LLIEN module, resulting in images with reasonable brightness. Additionally, this study proposes a novel intensity loss function to ensure that the fused image retains detailed information from the original pixel intensity. Extensive experimental results demonstrate that ILLVFusion has achieved significant performance in the image fusion task.