Multiscale Spatial-Spectral CNN-Transformer Network for Hyperspectral Image Super-Resolution

Abstract

Remarkable strides have been made in super-resolution methods based on deep learning for hyperspectral images (HSIs), which are capable of enhancing the spatial resolution. However, these methods predominantly focus on capturing local features using convolutional neural networks (CNNs), neglecting the comprehensive utilization of global spatial-spectral information. To address this limitation, we innovatively propose a multiscale spatial-spectral CNN-transformer network for hyperspectral image super resolution, namely, MSHSR. MSHSR not only applies the local spatial-spectral characteristics but also innovatively facilitates the collaborative exploration and application of spatial details and spectral data globally. Specifically, we first design a multiscale spatial-spectral fusion module, which integrates dilated-convolution parallel branches and a hybrid spectral attention mechanism to address the strong local correlations in HSIs, effectively capturing and fusing multiscale local spatial-spectral information. Furthermore, in order to fully exploit the global contextual consistency in HSIs, we introduce a sparse spectral transformer module. This module processes the previously obtained local spatial-spectral features, thoroughly exploring the elaborate global interrelationship and long-range dependencies among different spectral bands through a coarse-to-fine strategy. Extensive experimental results on three hyperspectral datasets demonstrate the superior performance of our method, outperforming comparison methods both in quantitative metrics and visual performance.