A Multimodal Semantic Segmentation Framework for Heterogeneous Optical and Complex SAR Data

Abstract

The advancement of remote sensing technology has led to a progressive enhancement in the resolution of remote sensing data, offering a multiperspective approach to Earth observation and facilitating a more comprehensive scene interpretation. As two most commonly utilized data sources in remote sensing, optical images, and synthetic aperture radar (SAR) data can provide complementary information, effectively compensating for the limitations inherent to a single modality. However, existing methods for using these two data sources face the following issues. First, insufficient utilization of the complete information provided by the source data. Second, inadequate consideration of the distinct characteristics of different modalities during feature extraction. Third, ignoring the misalignment between heterogeneous data, leading to large information loss. To tackle these challenges, we initially construct a benchmark dataset comprising complex-valued SAR data and optical images, named Multi-Complex-Seg. In order to fully mine the complete and valid information provided by both data sources, we construct a multimodal segmentation framework built on the theory of “subdomain extraction and cross-domain fusion,” in which we design a more suitable feature extractor for complex-valued SAR data, fully considering the unique geometric properties. In addition, a dynamic feature alignment module (DFAM) is proposed to further adjust the cross-modal features, and Cross-modal heterogeneous feature fusion module (CHFFM) first maps features into the same latent space to obtain better fused features. Both DFAM and CHFFM together reduce the huge semantic gap between modalities, thus facilitating the extraction of intramodal specificity and cross-modal complementarity. Extensive experiments on the proposed Multi-Complex-Seg confirm the effectiveness of our framework in comparison to other state-of-the-art multimodal segmentation approaches.