Beyond vision: A unified transformer with bidirectional attention for predicting driver perceived risk from multi-modal data

Modeling driver perceived risk (or subjective risk) plays a critical role in improving driving safety, as different drivers often perceive varying levels of risk under identical conditions, prompting adjustments in their driving behavior. Driving is a complex activity involving multiple cognitive and perceptual processes, such as visual information, driver feedback, vehicle dynamics, and traffic and environmental conditions. However, existing models for subjective risk perception have yet to fully address the need for integrating multi-modal data. To address this gap, we present a Transformer-based model aimed at processing multimodal inputs in a unified manner to enhance the prediction of subjective risk perception. Unlike existing methodologies that extract features specific to each modality, it employs embedding layers to transform images, unstructured, and structured fields into visual and text tokens. Subsequently, bi-directional multimodal attention blocks with inter-modal and intra-modal attention mechanisms capture comprehensive representations of traffic scene images, unstructured traffic scene descriptions, structured traffic data, environmental statistics, and demographics. Experimental results show that the proposed unified model achieves superior predictive performance over existing benchmarks while maintaining reasonable interpretability. Furthermore, the model is generalizable, making it applicable to various multi-modal prediction tasks across different transportation contexts.