Tensor Transformer for hyperspectral image classification

Abstract

Hyperspectral image (HSI) is widely used in real-world classification tasks since it contains rich spatial and spectral features consisting of hundreds of continuous bands. In recent years, the deep learning-based HSI classification methods, such as convolutional neural network (CNN) and Transformer, have achieved good performance in HSI classification tasks. Indeed, it is acknowledged that Transformer-based neural networks, owing to their remarkable capacity to extract long-range features, frequently outperform CNN-based neural networks in HSI classification scenarios. However, Transformer-based methods always require the sequentialization of the raw 3-D HSI data, potentially disrupting the spatial–spectral structural features. This shortcoming has degraded the classification accuracy of HSI data. In this paper, we proposed a Tensor Transformer (TT) framework for HSI classification. The TT model is an end-to-end network that directly takes the raw HSI tensor data as the input sample, without the need for raw data sequentialization. The core component of the proposed framework is the Tensor Self-Attention Mechanism (TSAM), which enables the network to efficiently extract long-range spatial–spectral structural features without losing the inherent structural relationships inner the sample. Through extensive experiments on four widely used HSI datasets, the proposed TT model demonstrates superior classification performance in discriminating land features with similar spectrum compared to state-of-the-art methods.