HSACT: A hierarchical semantic-aware CNN-Transformer for remote sensing image spectral super-resolution

Abstract

Hyperspectral remote sensing technology has demonstrated its spectral diagnosis advantages in numerous remote sensing observation fields. However, hyperspectral imaging is expensive and less portable compared to RGB imaging. To recover the corresponding hyperspectral image (HSI) from a remote sensing RGB image, this paper proposes a new hierarchical semantic-aware convolutional neural network (CNN)-Transformer (HSACT) for remote sensing image spectral super-resolution (SSR). Particularly, this work aims to reconstruct HSIs from RGB images within the same field of view using a lightweight semantic embedding architecture. Our HSACT consists of the following steps. First, an initial spectrum estimation module (from the RGB image to the HSI) is designed to progressively consider spectral estimation between RGB wavelength-inner and wavelength-outer information. Then, an attention-driven semantic-aware CNN-Transformer is developed to reconstruct the spatial and spectral details of HSI. Specifically, a trainable polymorphic superpixel convolution (PSConv) is proposed to capture features efficiently in the above module. Next, we introduce an information-lossless hierarchical network architecture to link the above modules and achieve end-to-end RGB image SSR through weight sharing. Experimental results on several datasets demonstrated that our HSACT outperforms traditional and advanced SSR methods. The codes of this paper are available from https://github.com/chengle-zhou/HSACT.