Human-Inspired Graphformer-Tutor: Accelerating and Safeguarding Multi-Agent Driving Learning

Abstract

Realistic modeling of interactive behaviors among multiple vehicles is essential for autonomous driving simulation. While Multi-Agent Reinforcement Learning (MARL) provides a promising framework for this task, the pure trial-and-error exploration paradigm often leads to inefficient and biased learning processes, limiting its ability to reproduce realistic driving behaviors. In light of this, inspired by the Tutor–Students learning paradigm and leveraging human expertise, a novel MARL algorithm termed Tutor–Multiple-Students Proximal Policy Optimization (TMSPPO) is proposed. A Graphformer-based tutor agent is developed to capture spatio-temporal driving features from human data and provide anthropomorphic assistance. TMSPPO embeds this tutor agent into the MARL loop through two complementary mechanisms: policy guidance and safety protection. Specifically, the guidance mechanism prevents student agents from cold-starting by integrating seamlessly with exploration, while the safety protection mechanism filters extreme samples yet retains informative near-misses to capture corrective data in near-danger scenarios. To further achieve a principled trade-off between autonomous exploration and tutor dependence, the intervention frequency and intensity are formulated as constrained objectives co-optimized with the learning goal. Consequently, TMSPPO substantially enhances learning efficiency and interactive driving capability, enabling student agents to display diverse and human-like behaviors in complex scenarios.