A visual–tactile fusion system for terrain perception under varying illumination conditions

Abstract

Road terrain conditions are vital for ensuring the driving safety of autonomous vehicles (AVs). However, traditional sensors like cameras and LiDARs are sensitive to changes in lighting and weather, posing challenges for real-time road condition perception. In this paper, we propose an illumination-aware visual–tactile fusion system (IVTF) for terrain perception, integrating visual and tactile data while optimizing the fusion process based on illumination characteristics. The system employs a camera and an intelligent tire to capture visual and tactile data across various lighting conditions and vehicle speeds. Additionally, we also design a visual–tactile fusion module that dynamically adjusts the weights of different modalities according to illumination features. Comparative results with single-modality perception methods demonstrate the superior ability of visual–tactile fusion to accurately perceive road terrains under diverse lighting conditions. This approach significantly advances the robustness and reliability of terrain perception in AVs, contributing to enhanced driving safety.