Modal-aware contrastive learning for hyperspectral and LiDAR classification

Abstract

Contrastive learning as a self-supervised learning method has received significant attention in the hyperspectral image (HSI) and light detection and ranging (LiDAR) data classification. However, the current contrastive learning-based methods ignore the huge gap between the HSI and LiDAR data in their ability to discriminate ground objects. To fully exploit the potential of HSI in the spectral domain and LiDAR in the spatial domain, we propose a modal-aware contrastive learning (MACL) framework, which learns discriminative multimodal features in both of spatial and spectral domains. First, we design a modal-aligned sample pair construction strategy to ensure that the data structure and characteristics of constructed spectral and spatial sample pairs remain consistent. Then, the spectral and spatial branches based on contrastive learning are adopted to extract multimodal spectral and spatial features in the pre-training stage. Finally, a multimodal attentional feature fusion (MAFF) module is designed to integrate and fuse the multimodal features for the downstream classification task, whose parameters are fine-tuned with a small number of labeled data. Experimental results on three public datasets, i.e., MUUFL, Trento, and Houston2013, demonstrate that our method outperforms several state-of-the-art methods in terms of qualitative and quantitative analysis. Our source codes are available at https://github.com/zlyrs1/MACL.