Interpretable Intersection Control by Reinforcement Learning Agent With Linear Function Approximator

Abstract

Reinforcement learning (RL) is a promising machine-learning solution to traffic signal control problems, which have been extensively studied. However, variants of non-linear, deep artificial neural network (ANN) function approximators (FAs) have been predominantly employed in previous studies proposing RL-based controllers, leaving a significant interpretability issue due to their black-box nature. In this work, the use of the linear FA for a value-based RL agent in traffic signal control problems is investigated along with the least-squares $Q$-learning method, abbreviated as $\mathrm{LSTD}Q$. The interpretable linear FA was found to be adequate for the RL agent to learn an optimal policy. This leads to the proposal to replace a non-linear ANN FA with the linear FA counterpart, resolving the interpretability issue. Moreover, the $\mathrm{LSTD}Q$ learning method shows superior behaviour convergence compared to a gradient descent method. In a low-intensity arrival pattern scenario, the control by the RL agent cuts about half of the average delay resulting from the pretimed control. Owing to the conciseness of the linear FA, a direct interpretation analysis of the converged linear-FA parameters is presented. Lastly, two online relearning tests of the agents under non-stationary arrivals are conducted to demonstrate the online performance of $\mathrm{LSTD}Q$. In conclusion, the linear-FA specification and the $\mathrm{LSTD}Q$ method are together proposed to be used for its control algorithm interpretability property, superior convergence quality, and lack of hyperparameters.