DSaC-ViT: Multi-scale guided upsampling fusion and parallel fusion vision transformer for hyperspectral image classification

Abstract

Hyperspectral images (HSI) capture rich spectral information for accurate land-cover classification. Recently, models based on hybrid architectures of convolutional neural networks (CNNs) and Transformers have been widely utilized for hyperspectral classification. However, a significant challenge is fully integrating the local features from CNN with the global features from Transformers. To alleviate this problem, we proposed an upsampling dual-scale fusion and self-attention convolutional parallel fusion vision Transformer (DSaC-ViT), which consists of a parallel self-attention convolutional vision Transformer (PSCViT) and a plug-and-play multi-scale guided upsampling feature fusion module (MGUFFM). PSCViT integrates the convolution and self-attention modules in parallel. Interacting between different patches via global token obtains global information representation. Adaptive parameters are then utilized to fuse this representation with local information extracted by CNN, thereby achieving granularity alignment. PSCViT can effectively extract and fuse local and global features. MGUFFM extracts spatial-spectral guidance features via a dual-branch structure to guide the upsampling fusion of high-level feature maps. This process effectively recovers missing spatial and spectral information. Four representative HSI datasets, encompassing agricultural, forest, urban, and wetland, were utilized in our extensive experiments. The results indicate that our proposed model outperforms other classification methods for HSI classification.