AdaptVFMs-RSCD: Advancing Remote Sensing Change Detection from binary to semantic with SAM and CLIP

Abstract

Remote Sensing Change Detection (RSCD) is essential for identifying surface changes from remote sensing images (RSIs) and plays a crucial role in land-use planning and disaster assessment. Despite advancements in RSI resolution and AI, most RSCD datasets are binary, hindering the transition to semantic change detection. Vision Foundation Models (VFMs), such as the Segment Anything Model (SAM), introduce new possibilities with robust zero-shot semantic segmentation capabilities, but face challenges with RSIs due to their unique characteristics, such as diverse perspectives and scale variations. To address these challenges, an enhanced RSCD method, AdaptVFMs-RSCD, has been proposed. This method integrates SAM with Contrastive Language-Image Pre-training (CLIP), capitalizing on CLIP’s ability to establish broad correspondences between images and text and to classify unseen image categories. This integration markedly improves the recognition of land cover types, better aligning VFMs with the specific requirements of RSCD. Additionally, a remote sensing VFM fine-tuning dataset was also developed to further enhance SAM’s segmentation performance on RSIs. Furthermore, a semantic information-based change detection module was designed to fully leverage both the change information and semantic information provided by VFMs, achieving state-of-the-art F1 and mIoU scores on the DSIFN (66.94%, 67.84%), CLCD (76.12%, 78.80%), and SYSU (81.14%, 78.35%) datasets. Notably, the comprehensive metrics F1 and mIoU on the SYSU dataset exceeded the second-best scores by 17.57% and 17.77%, respectively. AdaptVFMs-RSCD also facilitates the conversion of binary change detection datasets into semantic change detection datasets, advancing semantic change detection and expanding the application of vision language models and VFMs in remote sensing.