Improving Autonomous Vehicle Cognitive Robustness in Extreme Weather With Deep Learning and Thermal Camera Fusion

In autonomous vehicles (AV), sensor fusion methods have proven to be effective in merging data from multiple sensors and enhancing their perception capabilities. In the context of sensor fusion, the distinct strengths of multi-sensors, such as LiDAR, RGB, Thermal sensors, etc., can be leveraged to mitigate the impact of challenges imposed by extreme weather conditions. In this paper, we address multi-sensor fusion in AVs and present a comprehensive integration of a thermal sensor aimed at enhancing the cognitive robustness of AVs. Thermal sensors possess an impressive capability to detect objects and hazards that may be imperceptible to traditional visible light sensors. When integrated with RGB and LiDAR sensors, the thermal sensor becomes highly beneficial for detecting and locating objects in adverse weather conditions. The proposed deep learning-assisted multi-sensor fusion technique consists of two parts: (1) visual information fusion and (2) object detection using LiDAR, RGB, and Thermal sensors. The visual fusion framework employs a CNN (convolutional neural network) inspired by a domain image fusion algorithm. The object detection framework uses the modified version of the YoloV8 model, which exhibits high accuracy in real-time detection. In the YoloV8 model, we adjusted the network architecture to incorporate additional convolutional layers and altered the loss function to enhance detection accuracy in foggy and rainy conditions. The proposed technique is effective and adaptable in challenging conditions, such as night or dark mode, smoke, and heavy rain. The experimental results of the proposed method demonstrate enhanced efficiency and cognitive robustness compared to state-of-the-art fusion and detection techniques. This is evident from tests conducted on two public datasets (FLIR and TarDAL) and one private dataset (CUHK).