TAHALight: A Temporal-Aware, Hybrid-Action Reinforcement Learning Approach for Traffic Signal Plan Generation

Abstract

Most existing deep reinforcement learning (DRL) methods for traffic signal control rely on a high-frequency, second-by-second phase switching paradigm. While this paradigm often demonstrates superior performance in simulations, it faces significant challenges in real-world implementation. Traffic management practitioners prefer a cycle-based control paradigm, which, however, presents two fundamental challenges: (1) the curse of dimensionality in the action space, and (2) the difficulty of representing traffic states effectively given the long control intervals. This paper introduces TAHALight, a Temporal-Aware, Hybrid-Action Reinforcement Learning approach, that overcomes these obstacles through dedicated designs for the action space and state representation. To tackle the dimensionality curse, we propose a hierarchical hybrid action space. In this framework, a high-level policy selects a discrete cycle length from a predefined set, while a low-level policy determines the corresponding splits. This decomposition significantly reduces the action space dimensionality, thereby enhancing learning efficiency. To address the state representation challenge, we devise a state observation mechanism based on multi-period sampling and employ a Transformer-based state encoder. This encoder leverages a multi-head attention mechanism to model the spatio-temporal dependencies of traffic flow, enabling proactive and context-aware decision-making. Comprehensive 24-hour experiments conducted on both synthetic and real-world scenarios demonstrate that TAHALight can dynamically adapt to traffic flow variations. It significantly outperforms baseline methods in the synergistic optimization of cycle length and splits, reducing average delay and queue length by over 10% compared to both fixed-time control and state-of-the-art DRL baselines.