PAMSNet: A point annotation-driven multi-source network for remote sensing semantic segmentation

Abstract

Multi-source data semantic segmentation has proven to be an effective means of improving classification accuracy in remote sensing. With the rapid development of deep learning, the demand for large amounts of high-quality labeled samples has become a major bottleneck, limiting the broader application of these techniques. Weakly supervised learning has attracted increasing attention by reducing annotation costs. However, existing weakly supervised methods often suffer from limited accuracy. Effectively exploiting complementary information from multi-source remote sensing data using only a small number of labeled points remains a significant challenge. In this paper, we propose a novel architecture, named Point Annotation- Driven Multi-source Segmentation Network (PAMSNet), which leverages point annotations to effectively capture and integrate complementary features from multi-source remote sensing data. PAMSNet includes a Multi-source Feature Encoder and a Cross-Resolution Feature Integration (CRFI) module. The Multi-source Feature Encoder captures complementary global and local features using lightweight convolutional Global-Local Multi-source (GLMS) modules. And the boundary and spectral detail robustness are improved through Spectral-Edge Enhancement (SEE) modules, which effectively mitigate the impact of noise on segmentation accuracy. The CRFI module replaces conventional decoding structures by combining convolutional and Transformer mechanisms, enabling efficient cross-scale feature integration and improving the ability to identify multi-scale objects with reduced computational demands. Extensive experiments on the Vaihingen, WHU-IS, and WHU-OPT-SAR datasets validate the effectiveness of PAMSNet for weakly supervised multi-source segmentation as well as the validity of the proposed module. PAMSNet achieves state-of-the-art performance, with MIoU improvements of 2.4%, 2.1%, and 3.16% on three datasets, using only 0.01% point annotations. Additionally, PAMSNet can effectively balance the performance as well as the operational efficiency of the model compared to existing methods, which further promotes the application of deep learning in remote sensing image mapping.