Improved multi-agent deep reinforcement learning-based integrated control for mixed traffic flow in a freeway corridor with multiple bottlenecks

Abstract

A major challenging issue related to the emerging mixed traffic flow system, composed of Connected and Automated Vehicles (CAVs) and Human-Driven Vehicles (HDVs), is the lack of adequate traffic control measures, especially in a large freeway corridor with multiple bottlenecks. Multi-agent deep reinforcement learning exhibits significant advantages, such as fast response, high flexibility, strong adaptability, low computational burden, and collaborative optimization. These features enable it to achieve superior efficiency and robustness in handling dynamically changing traffic environments and large-scale traffic control problems. Inspired by this, we propose a novel Integrated Traffic Control (ITC) strategy based on an Improved Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (IPMATD3) algorithm in the mixed traffic environment (abbreviated as IPMATD3-based ITC). Specifically, the proposed IPMATD3-based ITC approach seeks to coordinate multiple Ramp Metering (RM) and Variable Speed Limit (VSL) controllers along a freeway corridor, with the objectives of improving traffic mobility and efficiency, enhancing safety, and reducing emissions. The proposed method utilized a centralized training with decentralized execution paradigm to learn the joint actions of all traffic controllers in a high-dimensional state and action spaces. A hybrid reward function is developed by synchronously considering the above objectives to optimize traffic control performance. Then, the rank-based prioritized experience replay mechanism is incorporated into the conventional MATD3 algorithm to improve learning efficiency. A real-world freeway corridor is selected to test the proposed control method. Moreover, its performance is compared with the several state-of-the-art methods. The simulation results demonstrate that the proposed method achieves remarkable control performance at a 10% CAV Penetration Rate (PR), effectively reducing the spatiotemporal extent of freeway traffic congestion. The proposed method outperforms other approaches in improving freeway traffic efficiency, mobility, safety, and environmental sustainability. Increasing the PR can improve the performance of various methods and benefit traffic operations. However, when the PR reaches higher levels, the marginal benefits of further increases become less pronounced.