IGFNet: Illumination-Guided Fusion Network for Semantic Scene Understanding using RGB-Thermal Images

Abstract

Semantic scene understanding is a fundamental task for autonomous driving. It serves as a build block for many downstream tasks. Under challenging illumination conditions, thermal images can provide complementary information for RGB images. Many multi-modal fusion networks have been proposed using RGB-Thermal data for semantic scene understanding. However, current state-of-the-art methods simply use networks to fuse features on multi-modality inexplicably, rather than designing a fusion method based on the intrinsic characteristics of RGB images and thermal images. To address this issue, we propose IGFNet, an illumination-guided fusion network for RGB-Thermal semantic scene understanding, which utilizes a weight mask generated by the illumination estimation module to weight the RGB and thermal feature maps at different stages. Experimental results show that our network outperforms the state-of-the-art methods on the MFNet dataset. Our code is available at: https://github.com/lab-sun/IGFNet.