Attribute-guided feature fusion network with knowledge-inspired attention mechanism for multi-source remote sensing classification

Abstract

Land use and land cover (LULC) classification is a popular research area in remote sensing. The information of single-modal data is insufficient for accurate classification, especially in complex scenes, while the complementarity of multi-modal data such as hyperspectral images (HSIs) and light detection and ranging (LiDAR) data could effectively improve classification performance. The attention mechanism has recently been widely used in multi-modal LULC classification methods to achieve better feature representation. However, the knowledge of data is insufficiently considered in these methods, such as spectral mixture in HSIs and inconsistent spatial scales of different categories in LiDAR data. Moreover, multi-modal features contain different physical attributes, HSI features can represent spectral information of several channels while LiDAR features focus on elevation information at the spatial dimension. Ignoring these attributes, feature fusion may introduce redundant information and effect detrimentally on classification. In this paper, we propose an attribute-guided feature fusion network with knowledge-inspired attention mechanisms, named AFNKA. Focusing on the spectral characteristics of HSI and elevation information of LiDAR data, we design the knowledge-inspired attention mechanism to explore enhanced features. Especially, a novel adaptive cosine estimator (ACE) based attention module is presented to learn features with more discriminability, which adequately utilizes the spatial–spectral correlation of HSI mixed pixels. In the fusion stage, two novel attribute-guided fusion modules are developed to selectively aggregate multi-modal features, which sufficiently exploit the correlations between the spatial–spectral property of HSI features and the spatial-elevation property of LiDAR features. Experimental results on several multi-source datasets quantitatively indicate that the proposed AFNKA significantly outperforms the state-of-the-art methods.