CloudRuler: Rule-based transformer for cloud removal in Landsat images

Abstract

Clouds are a key factor influencing transmission of the radiance signal in optical remote sensing images. For mapping or monitoring the Earth's surface, it is inevitable to mask or remove clouds before applying optical remote sensing images. Nowadays, deep learning (DL) based thin cloud removal methods far outperform traditional methods. Yet these DL-based methods often overlook position information or the physical cloud model in thermal bands. Moreover, most existing cloud physical models for cloud removal overlook the down-transmittance of the cloud in optical bands and do not account for the radiance of thermal bands. This work proposes a novel transformer network, CloudRuler, coupled with three rules in remote sensing domain for cloud removal. The proposed CloudRuler can distinguish the semantic meanings between similar features in different pixel positions by utilizing the Half-Spherical Coordinate System, aggregating features from local neighborhood windows with remote sensing mosaicking, and solving the parameters of the cloud physical model without limitations. Experimental results on 20 paired Landsat 8 and 9 images demonstrate that CloudRuler outperforms seven baseline methods, based on GAN, CNN, and transformer, both visually and quantitatively. Ablation experiments demonstrate that the proposed rule-based modules are highly effective in improving CloudRuler's performance for thin cloud removal. This work demonstrates that the joint use of Landsat 8 and 9 images for cloud removal is effective, producing more reliable data for downstream applications than methods that utilize only one satellite with a longer revisit period. For future research of the field, the code and dataset for reproducing the reported results are available on: https://github.com/Neooolee/CloudRuler.