CFFormer: A Cross-Fusion Transformer Framework for the Semantic Segmentation of Multisource Remote Sensing Images

Multisource remote sensing images (RSIs) can capture the complementary information of ground objects for use in semantic segmentation. However, there can be inconsistency and interference noise among the multimodal data from different sensors. Therefore, it is a challenge to effectively reduce the differences and noise between the different modalities and fully utilize their complementary features. In this article, we propose a universal cross-fusion transformer framework (CFFormer) for the semantic segmentation of multisource RSIs, adopting a parallel dual-stream structure to extract features separately from the different modalities. We introduce a feature correction module (FCM) that corrects the features of the current modality by combining features from the other modalities in both the spatial and channel dimensions. In the feature fusion module (FFM), we employ a multihead cross-attention mechanism to interact globally and fuse features from the different modalities, enabling the comprehensive utilization of the complementary information in multisource RSIs. Finally, comparative experiments demonstrate that the proposed CFFormer framework not only achieves state-of-the-art (SOTA) accuracy but also exhibits outstanding robustness when compared to the current advanced networks for semantic segmentation of multisource RSIs. Specifically, CFFormer achieves a mean intersection over union (mIoU) of 58% and an overall accuracy (OA) of 85.35% on the WHU-OPT-SAR dataset, outperforming the second-ranked network by 4.71% and 1.74%, respectively. On the Vaihingen and Potsdam datasets, CFFormer also achieves the best results, with mIoU and OA values of 84.31%/91.88% and 88.62%/92.64%, respectively. The source code is available at https://github.com/masurq/CFFormer .