AM2CFN: Assimilation Modality Mapping Guided Crossmodal Fusion Network for HSI and LiDAR Data Joint Classification

Abstract

Combining their complementary properties, using hyperspectral image (HSI) and light detection and ranging (LiDAR) data improves classification performance. Nevertheless, the heterogeneous capturing instruments and distribution characteristics of these two remote sensing (RS) modalities always limit their application scopes in on-ground observation-related domains. This heterogeneity hinders capturing the crossmodal connection for discriminant information extraction and exchange. In this letter, we propose an assimilation modality mapping guided crossmodal fusion network (AM2CFN) for HSI and LiDAR data joint classification. Our motivation is to explore one RS assimilation modality (RSAM) by exploiting one latent crossmodal mapping strategy from HSI and LiDAR data simultaneously to remove the effect of modality heterogeneity and contribute to information exchange. AM2CFN constructs one level-wise assimilating encoder to simulate modality heterogeneity and enhance regional consistency. Modality intrinsic features are captured in this encoder to provide knowledge for modality assimilation. Furthermore, one RSAM balancing HS and LiDAR properties is explored. AM2CFN constructs one RSAM reconstruction decoder for modality reconstruction and classification. Dual constraints based on solid angle and Kullback-Leibler divergence are considered to restrain the information exchange process toward the optimal direction. Experiments show that AM2CFN outperforms several state-of-the-art techniques qualitatively and quantitatively. AM2CFN increases the overall accuracy (OA) by 2.46% and 1.62% on average on the Houston and MUUFL datasets. The codes will be available at https://github.com/GEOywb/AM2CFN