MBSSNet: A Mamba-Based Joint Semantic Segmentation Network for Optical and SAR Images

Abstract

The utilization of both optical and synthetic aperture radar (SAR) images for joint semantic segmentation enhances the accuracy of land use classification. Recent advancements in multimodal fusion models, particularly those using self-attention mechanisms and convolutional neural networks (CNNs), have yielded significant results. However, self-attention has quadratic computational complexity, and CNN has insufficient local-global contextual modeling power. Recently, 2-D-selective-scan (SS2D) has emerged as a promising approach. It excels in modeling long-range dependencies while maintaining linear computational complexity. Based on SS2D, we propose a joint semantic segmentation network for optical and SAR images, called MBSSNet. Specifically, we introduce SS2D and design a cross-modal fusion module (CMFM) to fuse multimodal features from dual branches layer by layer, thereby enhancing the consistency of fused feature representations. In addition, during the decoding phase, we integrate contextual information from multiscale fusion features, thereby enhancing the spatial and semantic information of the fused features. Our experimental results show that our method outperforms the state-of-the-art (SOTA), and overall accuracy (OA), mean intersection over union (mIoU), and Kappa outperform other SOTA methods by 1.7%, 3.1%, and 2.2%, respectively.