Spectral–Spatial Feature Extraction Network With SSM–CNN for Hyperspectral–Multispectral Image Collaborative Classification

Abstract

Multisource remote sensing (RS) image classification is a significant research area in Earth observation, aiming to achieve more comprehensive and accurate classification of land cover by integrating data from different sensors. Due to differences in imaging mechanisms and information imbalance between multisource data, multisource RS image classification faces two major challenges as follows. 1) Synergistically capturing features from different modalities to fully exploit complementary information. 2) Adaptively fusing multisource features to overcome the imbalance between modalities and avoid redundant information. This article proposes a spectral–spatial feature extraction network with SSM–CNN (SSFNet) for the collaborative classification of hyperspectral images (HSI) and multispectral images (MSI). Specifically, SSFNet captures long-range spectral correlations in HSI through a bidirectional state–space model (SSM) and learns local correlations between adjacent channels through spectral grouping, achieving global–local spectral information mining in HSI. Simultaneously, joint spatial feature extraction for HSI and MSI data is performed using embedded weight-shared residual feature extractor based on convolutional neural network. This process involves adaptively identifying the importance of features through privatized factors in batch normalization and accurately replacing redundant features. In addition, a spatial attention module is used to further enhance spatial feature representation. Finally, to better accommodate feature distributions and enhance classification outcomes, the extracted spectral–spatial features are combined using weighted fusion, allowing for dynamic integration. Experimental results on two datasets demonstrate that the proposed SSFNet significantly outperforms other competing methods.