Fusing Sentinel-1 and Sentinel-2 data with diffusion models for cloud removal

Abstract

Cloud cover significantly hinders the use of optical remote sensing data, such as Sentinel-2, by obscuring critical information needed for environmental monitoring. This study introduces Enhanced Diffusion Model for Cloud Removal (EDM-CR), an enhanced cloud removal framework that integrates Sentinel-1 synthetic aperture radar (SAR) and Sentinel-2 optical imagery using efficient diffusion models. Our approach features three key novel aspects: (1) a forward diffusion process conditioned on both the previous timestep image and cloudy Sentinel-2 data, simulating cloud addition to improve the backward diffusion process; (2) a two-branch backward diffusion process conditioned on the current timestep image, cloudy Sentinel-2 data, and Sentinel-1 data, enhancing cloud removal fidelity and restoration efficiency; and (3) a modified Learned Perceptual Image Patch Similarity (LPIPS) loss function that incorporates all 13 Sentinel-2 spectral bands, ensuring comprehensive spatial information preservation. The framework is trained using paired and co-registered Sentinel-1 and Sentinel-2 images (both cloudy and cloud-free) from the SEN12MS-CR dataset and validated on a large set of unseen cloudy images. Experimental results demonstrate that our method outperforms five state-of-the-art cloud removal techniques. Furthermore, the cloud-removed Sentinel-2 images are used as year-round inputs for farmland segmentation in the Netherlands, providing temporal context that improves segmentation accuracy compared to using limited timesteps. These findings underscore the effectiveness of our diffusion model framework in integrating multi-sensor data for robust cloud removal and highlight the benefits of incorporating temporal information for accurate semantic segmentation of farmland.