SAM-CTMapper: Utilizing segment anything model and scale-aware mixed CNN-Transformer facilitates coastal wetland hyperspectral image classification

Abstract

Accurate and effective coastal wetland classification using hyperspectral remote sensing technology is crucial for their conservation, restoration, and sustainable development. However, the large scale variance of land covers in complex wetland scenes poses challenges for existing methods and leads to misclassifications. Additionally, existing methods encounter difficulties in practical wetland classification tasks due to the high cost of hyperspectral wetland data labeling. This paper introduces SAM-CTMapper, a coastal wetland classification framework that incorporates a scale-aware mixed CNN-Transformer (CTMapper) to precisely identify wetland cover types using hyperspectral images, and the advanced segment anything model (SAM) to save labor costs in data labeling. Specifically, a novel scale-aware mixed CNN-Transformer layer is designed in CTMapper to effectively leverage local and long-range spectral–spatial features from the whole HSI to reduce misclassification. This layer comprises a multi-head scale-aware convolution layer to capture local land-cover details, a multi-head superpixel self-attention layer for extracting long-range contextual features, and a dynamic selective module to facilitate effective aggregation of local and long-range information. Additionally, we devise a SAM-based semi-automatic labeling strategy to construct two PRISMA hyperspectral wetland (PRISMA-HW) datasets over Liaoning Shuangtai and Shanghai Chongming for evaluation purposes. Experimental results on two PRISMA-HW datasets and two publicly available hyperspectral wetland datasets demonstrate the effectiveness of CTMapper method in terms of both accuracy metrics and visual quality. For the sake of reproducibility, the PRISMA-HW datasets and the related codes of SAM-CTMapper framework will be open-sourced at: https://github.com/immortal13.