Replacing complex transformer with simple attention to achieve hyperspectral and multispectral image fusion

The fusion of hyperspectral (HSI) and multispectral (MSI) images to obtain high-resolution hyperspectral images is crucial for hyperspectral image processing and interpretation. In recent years, the Transformer architecture is extensively utilized in the domain of HSI-MSI fusion, yielding promising results. However, the introduction of various modified Transformer architectures leads to increasingly complex network structures, which impose significant computational resource demands. To address this issue, this paper proposes a Mask and Cross-Attention Network (MCANet) for HSI and MSI fusion. The network comprises three components: masked feature extraction, channel and spatial attention cross-feature fusion, and multi-scale step-by-step reconstruction. Our network abandons the existing patchwork of various advanced attention mechanisms and instead employs only the most straightforward channel spectrum and spatial attention mechanisms. This approach allows us to thoroughly extract spectral and spatial features while minimizing the computational resources required by the model. We conduct experiments on 8 HSI datasets and compare them with state-of-the-art fusion methods. The Pavia Center, Pavia University, Washington DC, Salinas, Houston and Botswana datasets all achieve the best fusion images and metrics. In addition, efficiency experiments confirm that the proposed method saves a significant amount of computational resources while fusing high-quality images. The source code and pre-trained models are available at https://github.com/xiaomudsg/MCANet.