Safe Multi-Agent Deep Reinforcement Learning for the Management of Autonomous Connected Vehicles at Future Intersections

Abstract

As Connected and Autonomous Vehicles (vehicle) evolve, Autonomous Intersection Management (AIM) systems are emerging to enable safe, efficient traffic flow at urban intersections without traffic signals. However, existing AIM systems, whether based on traditional optimization control methods or machine learning, suffer from low computational efficiency and a lack of robustness in ensuring safety, respectively. To overcome these limitations, we propose an innovative AIM scheme rooted in Safe Multi-Agent Deep Reinforcement Learning (MADRL). We initially model the safe MADRL problem as a constrained Markov game (CMG) and tackle it with our multi-agent projective constrained policy optimization (MAPCPO). This method first optimizes policy updates within the Kullback-Leibler divergence trust region to maximize performance, and then projects these optimized policies onto the bounds of risk constraints, thus ensuring safety. Building on this, we introduce a Risk-Bounded RL for Autonomous Intersection Management (RbRL-AIM) algorithm. This algorithm adopts an architecture that consists of an LSTM based policy neural network, a reward value network, and a risk neural network. These components, through the MAPCPO policy, enable continuous learning from complex and random intersection traffic environments, thereby facilitating the safe, efficient, and smooth control of vehicles at intersections. Our method is validated in a CARLA simulation, showing significant gains in computational and traffic efficiency over baseline optimization control methods. Compared to non-safety-aware MADRL methods, our approach achieves zero collisions and improved ride comfort.