Cloud removal with optical and SAR imagery via multimodal similarity attention

Abstract

Optical remote sensing images are crucial data sources for various applications, including agricultural monitoring, land cover classification, and urban planning. However, cloud cover often hinders their effectiveness, which poses a significant challenge to downstream tasks. To address this issue, we introduce the Similarity-based Multimodal De-Clouding Network (SMDCNet), an innovative framework that enhances the quality of optical remote sensing images by utilizing multimodal similarity attention to integrate complementary information from synthetic aperture radar (SAR) imagery. First, we introduce a similarity feature attention (SFA) module that explores the similarity between optical and SAR features, aligning these cross-domain features to guide the optical encoder’s focus on cloud-free regions for more accurate feature alignment. Building on this, we propose a differential feature extraction (DFE) module that selectively uses SAR features to compensate for cloud-covered regions in the optical images. To mitigate the blurriness in the de-clouded images, we incorporate differential characteristics injection (DCI) and multi-scale feature fusion (MSFF) modules, which collaboratively enhance the reconstruction of detailed information. Our experiments on the SEN12MS-CR dataset demonstrate that SMDCNet effectively restores high-quality cloud-free images, achieving a PSNR of 30.2759 dB, outperforming state-of-the-art cloud removal techniques.