Bridging phase and timing: A joint Q-value learning framework for synergistic traffic signal control at consecutive arterial road intersections

Abstract

Reinforcement Learning (RL) is a highly effective traffic control method and shows strong potential. However, existing reinforcement learning-based methods for traffic signal control are insufficient in state modeling and cannot distinguish between the direct impact of queued vehicles on the intersection and the dynamic impact of moving vehicles. Additionally, they lack the ability to capture the deep synergistic relationship between signal phases and dynamic phase durations, which often leads to suboptimal decisions. This paper proposes an RL traffic signal control method based on joint Q-values, employing a detailed and predictive state representation that fully considers the different effects of queued and moving vehicles on intersection congestion. By combining vehicle speed and position features, the dynamic influence of moving vehicles on future traffic pressure is quantified. Meanwhile, the model effectively integrates lane and phase features using a multi-head attention mechanism, automatically capturing the conflict and cooperation relationships among different traffic flows. On this basis, a joint Q-value learning framework is adopted, treating signal phase selection and dynamic duration decisions as a complete decision unit for joint optimization. This directly learns the synergy between the two, thereby avoiding the problem of suboptimal decisions where efficient phases are chosen but with insufficient timing. Comprehensive experimental results in both real and synthetic scenarios show that our method achieves up to a 7.20% reduction in average travel time across various intersection settings, while also having a higher vehicle throughput. Moreover, the model converges to the optimal solution more quickly. This characteristic ensures excellent model performance while maintaining strong generalization capability.