3CNet: Cross-modal cooperative correction network for RGB-T semantic segmentation

Utilizing RGB-Thermal (RGB-T) data, multi-modal semantic segmentation enables pixel-wise classification of images across diverse environmental conditions, including variations in lighting and occlusions. However, a significant challenge persists in reducing the discrepancies between modalities while effectively utilizing their complementary strengths. To address this issue, a novel Cross-modal Cooperative Correction Network (3CNet) is proposed for RGB-T semantic segmentation. The core of 3CNet lies in the “correction-then-fusion” strategy. The cross-modal cooperative correction module employs orthogonal and spatial attention mechanisms to rectify the feature representations, thereby ensuring consistency and reliability of features for subsequent fusion. Additionally, a unique triple-stream feature fusion structure is introduced to enhance the efficiency of multi-modal feature utilization and improve overall fusion performance. Our proposed network demonstrates state-of-the-art performance on two RGB-T datasets, highlighting the potential of multi-modal information in advancing segmentation accuracy and confirming its efficacy in practical applications. The code and results are available at https://github.com/GraceGuoo/3CNet.