SAR-to-Optical Image Translation Using Vision Transformer-Based CGAN

Abstract

In this article, we propose a modified conditional generative adversarial network (CGAN) for synthetic aperture radar (SAR)-to-optical image translation. SAR imaging is widely employed in remote sensing due to its all-weather capability, yet its inability to capture color information can limit the extraction of crucial details. Consequently, several attempts aim to convert SAR images into optical images to overcome these shortcomings. Our research adopts a multilevel approach that jointly incorporates high-level and low-level terrain extracted from SAR images and color features. Our proposed method is composed of multiscale vision Transformer (viT) blocks, enhanced loss function based on perceptual differences, and a two-phase transfer learning technique to guide the network properly. First, we obtain the multiaspect features of SAR images by using the intermediate viT processor and viT bottleneck. Second, we add a perceptual loss function that is driven from the pretrained VGG-19 network to overcome the blurring nature of traditional L1 loss. Finally, we employ a two-phase transfer learning using patched grayscale optical images and noise-removed images pre-processed by the Lucy-Richardson filter. Compared to previous studies that use only a small portion of the SEN1-2 dataset under 20000 image pairs, our model is trained on 75724 image pairs from this dataset. It ensures a significantly broader and more representative coverage of conditions and scenes compared to previous studies. The output image of our proposed method stands out with the highest peak signal-to-noise ratio (PSNR) of 15.96 dB and structural similarity index measure (SSIM) of 0.2805, along with the lowest mean-squared error (mse) 0.0363 and Fréchet inception distance (FID) score of 142.1333, proving that it generates superior images compared to conventional models in all respects.