A deep learning and image enhancement based pipeline for infrared and visible image fusion

It is difficult to use supervised machine-learning methods for infrared (IR) and visible (VIS) image fusion (IVF) because of the shortage of ground-truth target fusion images, and image quality and contrast control are rarely considered in existing IVF methods. In this study, we proposed a simple IVF pipeline that converts the IVF problem into a supervised binary classification problem (sharp vs. blur) and uses image enhancement techniques to improve the image quality in three locations in the pipeline. We took a biological vision consistent assumption that the sharp region contains more useful information than the blurred region. A deep binary classifier based on a convolutional neural network (CNN) was designed to compare the sharpness of the infrared region and visible regions. The output score map of the deep classifier was treated as a weight map in the weighted average fusion rule. The proposed deep binary classifier was trained using natural visible images from the MS COCO dataset, rather than images from the IVF domain (called “cross domain training” here). Specifically, our proposed pipeline contains four stages: (1) enhancing the IR and VIS input images by linear transformation and the High-Dynamic-Range Compression (HDRC) method, respectively; (2) inputting the enhanced IR and VIS images to the trained CNN classifier to obtain the weight map; and (3) using a weight map to obtain the weighted average of the enhanced IR and VIS images; and (4) using single scale Retinex (SSR) to enhance the fused image to obtain the final enhanced fusion image. Extensive experimental results on public IVF datasets demonstrate the superior performance of our proposed approach over other state-of-the-art methods in terms of both subjective visual quality and 11 objective metrics. It was demonstrated that the complementary information between the infrared and visible images can be efficiently extracted using our proposed binary classifier, and the fused image quality is significantly improved. The source code is available at https://github.com/eyob12/Deep_infrared_and_visible_image_fusion.