Is cooperative always better? Multi-Agent Reinforcement Learning with explicit neighborhood backtracking for network-wide traffic signal control

Abstract

Multi-Agent Reinforcement Learning (MARL) has been empirically demonstrated as a highly promising paradigm for the Cooperative Traffic Signal Control (CTSC) of urban road networks. A review of recent MARL-based literature reveals a counter-intuitive finding: several sophisticated approaches have been outperformed by simpler independent control schemes when applied across multiple intersections. This paper analyzes the phenomenon and proposes a hypothesis that the setting of surveillance zone length may determine whether a MARL-based CTSC algorithm is effective or not. We prove this hypothesis qualitatively and quantitatively and find that the intersection interactions are time-lagged. Faced with the incomplete surveillance zone, we model the CTSC process as a decentralized partially observable Markov decision process (Dec-POMDP). Further, we propose ENB-RL, a MARL model with explicit neighborhood backtracking to handle the lag in impacts from neighbors. The core of our proposal is an ENB module, which consists of a neighborhood backtracking stack to store and update neighborhood intersections’ historical throughput in a segmented weighted way, and a multi-head attention model for spatio-temporal differentiated input. Such explicit and precise inputs can improve the agent’s observations in incomplete perceptual environments. Considering that historical backtracking information may lead to convergence instability, we introduce random Gaussian noise for Double Deep Q-Network (DDQN) to generate uncertainty and improve the efficiency and stability of exploration. Experimental results show that ENB-RL has the best convergence performance on both synthetic and real-world datasets, and outperforms other state-of-the-art MARL models. Ablation experiments confirm the efficacy of each component in the framework. Moreover, the proposed ENB module can also be plugged and played in mainstream RL-based models.