HCAFNet: Hierarchical Cross-Modal Attention Fusion Network for HSI and LiDAR Joint Classification

Abstract

Hyperspectral image (HSI) and light detection and ranging (LiDAR) data can provide complementary features and have shown great potential for land cover classification. Recently, the joint classification using the HSI and LiDAR data based on deep learning networks (e.g., convolutional neural networks and transformers) have made progress. However, these methods often use the channel or spatial dimension attentions to highlight features, which overlook the interdependencies between these dimensions and face challenges in effectively extracting and fusing diverse features from heterogeneous datasets. To address these challenges, a novel hierarchical cross-modal attention fusion network (HCAFNet) is proposed in this manuscript. First, a hierarchical convolution module is designed to extract diverse features from multisource data and to achieve initial fusion using the octave convolution. Then, a bidirectional feature fusion module is constructed to integrate heterogeneous features within the network. To further enhance the network's feature representation capability, a triplet rotational multihead attention module is designed to capture cross-dimensional dependencies, enabling more effective representation of both channel and spatial information. Experimental results conducted on three public datasets demonstrate that the proposed HCAFNet outperforms other advanced methods.