Distributed multi-agent reinforcement learning approach for energy-saving optimization under disturbance conditions

Abstract

Urban rail transit systems exhibit substantial energy consumption, underpinning the significance of energy-saving optimization strategies for train timetables. Conventionally, trains operate according to an energy-efficient timetable formulated offline. However, station incidents and disturbances often result in deviations from the planned schedule, leading to additional energy expenditure. To address this challenge, the current study introduces a distributed multi-agent reinforcement learning approach(DMARL) for real-time energy-efficient optimization of train timetables. Initially, trains are conceptualized as agents, adopting the Actor-Critic network structure as the learning paradigm, with a distributed deployment architecture facilitating the training of the model. During the interaction phase between agents and the subway system, a progressive reward mechanism is designed to encourage efficient exploratory actions by the agents. In the final case study, data from Shanghai Metro Line 1(SML1) was utilized to demonstrate the effectiveness of the proposed method. The results indicate that when disturbances occur at stations, necessitating extended stop times, the method presented in this paper exhibited stable performance and faster convergence rates in both two-train and three-train systems. Compared to the energy consumption without any action, the energy savings were enhanced by 14.11 % and 11 %, respectively. The timetable updates were completed within milliseconds, confirming the efficacy of the method and its compliance with real-time updating requirements.