A spatial–spectral fusion convolutional transformer network with contextual multi-head self-attention for hyperspectral image classification

Abstract

Convolutional neural networks (CNNs) can effectively extract local features, while Vision Transformer excels at capturing global features. Combining these two networks to enhance the classification performance of hyperspectral images (HSI) has garnered significant attention. However, most existing fusion methods introduce inductive biases for the Transformer by directly connecting convolutional modules and Transformer encoders for feature extraction but rarely enhance the Transformer’s ability to extract local contextual information through convolutional embedding. In this paper, we propose a spatial–spectral fusion convolutional Transformer method (SSFCT) with contextual multi-head self-attention (CMHSA) for HSI classification. Specifically, we first designed a local feature aggregation (LFA) module that utilizes a three-branch convolution architecture and attention layers to extract and enhance local spatial–spectral fusion features. Then, a novel CMHSA is built to extract interaction information of local contextual features through integrating static and dynamic local contextual representations from 3D convolution and attention mechanisms, and the CMHSA is integrated into the devised dual-branch spatial–spectral convolutional transformer (DSSCT) module to simultaneously capture global–local associations in both spatial and spectral domains. Finally, the attention feature fusion (AFF) module is proposed to fully obtain global–local spatial–spectral comprehensive features. Extensive experiments on five HSI datasets — Indian Pines, Salinas Valley, Houston2013, Botswana, and Yellow River Delta — outperform state-of-the-art methods, achieving overall accuracies of 98.03%, 99.68%, 98.65%, 97.97%, and 89.43%, respectively, showcasing its effectiveness for HSI classification.