Multimodal Large-Language Model Empowering Next-Generation Autonomous Driving Systems

Abstract

Autonomous driving technology has made significant advancements in recent years. The evolution of autonomous driving systems from traditional modular designs to end-to-end learning paradigms has led to comprehensive improvements in driving capabilities. In modular designs, driving tasks are segmented into independent modules, such as perception, decision-making, planning, and control. This modular structure offers high explainability and safety in simple scenarios but is hindered by limited generalizability in complex traffic environments, and the sequential connection of multiple modules often leads to error accumulation. In contrast, end-to-end methods process perception data directly to produce control outputs, thereby mitigating information loss and sequential error accumulation, ultimately improving scene generalization in diverse environments. However, this approach is limited by strong data dependency, low interpretability, and inadequate handling of long-tail scenarios (Zhao et al., 2024).