Multiscale Low-Rank and Sparse Attention-Based Transformer for Hyperspectral Image Classification

Abstract

Recently, transformer-based approaches have emerged as powerful tools for hyperspectral image (HSI) classification. HSI inherently exhibits low-rank and sparse properties due to spatial continuity and spectral redundancy. However, most existing methods directly adopt standard transformer architectures, overlooking the distinctive priors inherent in HSI, which limits the classification performance and modeling efficiency. To address these challenges, this letter proposes a multiscale low-rank and sparse transformer (MLSFormer) that effectively integrates both low-rank and sparse priors. Specifically, we leverage tensor low-rank decomposition (TLRD) to factorize the query, key, and value matrices into low-rank tensor products, capturing dominant low-rank structures. In parallel, we introduce a sparse attention mechanism to retain only the most important connections. Furthermore, a multiscale attention mechanism is designed to hierarchically partition attention heads into global, medium, and local groups, each assigned tailored decomposition ranks and sparsity ratios, enabling comprehensive multiscale feature extraction. Extensive experiments on three benchmark datasets demonstrate that MLSFormer achieves superior classification performance compared to state-of-the-art methods.