A multimodal traffic scene understanding model integrated with optical flow maps

Abstract

Autonomous driving technology has rapidly developed in recent years, promising wide-ranging applications in the transportation industry. However, the decision-making processes of autonomous driving systems are nontransparent and nonexplainable, hindering the broader adoption and industrial deployment of this technology. To overcome these challenges, this article proposes an end-to-end traffic scene understanding model (TSUM) integrated with various proven performance enhancement techniques derived from research on multimodal large models. First, features from Farneback optical flow maps and raw scene videos are simultaneously extracted using a shared-parameter network, which effectively captures the temporal dynamics of traffic scenes. By incorporating scene perception and causal analysis at the task layer, an adaptive task complexity progression strategy is designed, which reduces the task complexity. Finally, we establish a guiding library with a diverse array of guiding phrases to enhance the generalization capabilities of the TSUM. The model is experimentally validated on the publicly available Berkeley Deep Drive-X dataset. Comparison studies confirmed that the TSUM outperforms state-of-the-art methods across multiple evaluation metrics.