Multi spectral visible-thermal IR image translation using improved u-net & conditional diffusion

Abstract

Translating images from visible spectrum to thermal IR (TIR) domain to achieve precise and realistic representations of TIR images is a challenging task. Thermal infrared imaging is of great significance in scenarios where vision is severely impaired especially in difficult lighting conditions such as night, haze, fog or cloudy weather. With these advantages, infrared imaging finds extensive applicability in navigation, surveillance, object detection, product inspection, agriculture as well as remote sensing. In order to build high performance deep models for such wide range of applications, it is necessary to have large amount of TIR data for training. However, there is unavailability of sufficient IR based datasets due to high cost of thermal infrared camera setups. While large number of visible image datasets are available, this scarcity of TIR datasets can be addressed by translating visible images to their TIR counterparts. In this paper, we leverage the widely available visible range data to propose two visible to TIR domain translation approaches, one is modified U-Net based non-generative approach called TIR-UNet and the other is conditional diffusion based generative approach that also uses U-Net as neural backbone for synthesizing TIR images. Both the proposed methods have been evaluated on four benchmark datasets and demonstrate high qualitative as well as quantitative performance in generating perceptually realistic, visually plausible and high quality TIR equivalents of given visible images. Compared to state-of-the-art methods which include U-Net and powerful GAN variants, our methods achieve remarkable performance increase on the metrics of MSE, PSNR and SSIM for both day and night images.