FFSNet: Adaptive features fusion of foundation models and self-supervised models for remote sensing image segmentation

Abstract

Remote sensing image segmentation is essential for urban planning, environmental monitoring, and disaster assessment but is challenged by scarce pixel-level annotations, domain shifts, and the difficulty of segmenting spectrally similar land cover classes. Existing methods struggle to address these issues comprehensively. Supervised approaches like UNetFormer require extensive labeled data and have limited generalization. Foundation models like SAM enable zero-shot segmentation but are constrained by high inference overhead, limiting their practical use in remote sensing. Self-supervised models like DINO capture domain-specific features but lack the global priors and generalization capabilities of large-scale foundation models, reducing their effectiveness in complex remote sensing scenarios. To overcome these limitations, FFSNet is proposed as a novel framework that integrates a lightweight MobileSAM encoder with a DINOv2 self-supervised encoder pretrained on remote sensing data. Its core innovation, the adaptive feature fusion module, balances general visual priors and domain-specific representations using attention-based dynamic weighting. Additionally, a modified category mask decoder extends binary output to multi-class segmentation using learnable prototype vectors. Experiments on three benchmark datasets validate the effectiveness of FFSNet. It achieves a mIoU of 55.4 % on LoveDA, surpassing D2lS, a mF1 of 88.3 % on ISPRS Potsdam, outperforming AerialFormer, and a mF1 of 91.6 % on Vaihingen, while using only 44.7 M parameters—a 50 % reduction compared to D2lS. FFSNet establishes a new paradigm for efficient domain adaptation in foundation models, offering superior segmentation accuracy with reduced computational costs, making it highly practical for large-scale remote sensing applications.