MCU-GAN: Colorization method for infrared images based on multi-convolution fusion and generative adversarial network

Abstract

Since traditional cameras perform poorly in images acquired at night or in extreme weather, such as heavy fog, this limits their use in military, night-time traffic management, security monitoring and other fields. Infrared image sensor can overcome these limitations and present thermal infrared features in image. However, because thermal infrared (TIR) image is based on the thermal radiation emitted by the object itself, this will result in the infrared image lacking detailed information, such as texture and color, which is completely unsuitable for direct observation by human eye. Therefore, many thermal infrared image colorization methods have emerged in recent years, which can convert thermal infrared images into color images. However, the current infrared image colorization results still have flaws in semantics and color recovery. Thus, we propose a thermal infrared image colorization method based on multi-convolution fusion and generative adversarial network, called MCU-GAN. First, we designed a novel feature extraction module (MCRB) in the downsampling process of the generator. The module combines pixel difference convolution and dilation convolution, which effectively improves the ability to extract edge features of objects in TIR images. Second, we combine the ideas of ConvNeXt with the U-shaped neural network to form a generator, which enables the model to compete with Transformers in terms of performance while ensuring simplicity. Then, the polarized self-attention module (PSA) is added in the decoding stage to filter irrelevant feature information. Finally, we use a composite loss function to ensure that the network can generate color images that are discernible to human eye. Experimental results show that MCU-GAN achieves better results on the KAIST and IRVI-Traffic datasets.