A heterogeneous agent reinforcement learning approach with curriculum learning for variable speed limit control

Abstract

The majority of existing Variable Speed Limit (VSL) studies employ rule-based control strategies, which often fail to accommodate diverse road characteristics and lack responsiveness to sudden congestion. These limitations are further compounded by computational inefficiencies, hindering real-time decision-making in large-scale or complex traffic environments. To address these issues, this study proposes a VSL strategy based on Heterogeneous Agent Reinforcement Learning with Curriculum Learning (HARLCL). Specifically, a top-level agent classifies road segments using the Mini-Batch K-means algorithm, thereby capturing the heterogeneity of each segment. The lower-level agents operate with class-specific observation spaces and action spaces, each agent observes and acts its feature set tailored to its segment class while sharing the same reward design and learning architecture. Subsequently, lower-level agents adaptively adjust both the position and length of each controlled segment according to classification results and real-time traffic conditions. Through a reward-driven mechanism, these agents continually refine their control precision. Moreover, curriculum learning is introduced during the multi-agent training process, effectively accelerating convergence and mitigating computational burdens typically encountered in large-scale reinforcement learning. Experiments were conducted in three task scenarios–typical fixed bottlenecks, random task bottlenecks, and multiple bottlenecks using realistic road data. The results indicate that training time decreased by 43.18 % and 47.35 %; in the SUMO microscopic simulation, total travel time was 312.25 veh·h, 43.05 % lower than NoVSL, and safe braking distances also decreased, indicating improved safety and more stable traffic flow. Compared with conventional feedback control and deep reinforcement learning approaches, the HARLCL-based strategy demonstrates substantial advantages in training efficiency and control precision, offering a promising avenue for practical VSL implementation.