A Multi-Task Reinforcement Learning Approach for Optimal Sizing and Energy Management of Hybrid Electric Storage Systems Under Spatio-Temporal Urban Rail Traffic

Abstract

Passenger flow fluctuation and delay-induced traffic regulation bring considerable challenges to cost-efficient regenerative braking energy utilization of hybrid electric storage systems (HESSs) in urban rail traction networks. This paper proposes a synergistic HESS sizing and energy management optimization framework based on multi-task reinforcement learning (MTRL) for enhancing the economic operation of HESSs under dynamic spatio-temporal urban rail traffic. The configuration-specific HESS control problem under various spatio-temporal traction load distributions is formulated as a multi-task Markov decision process (MTMDP), and an iterative sizing optimization approach considering daily service patterns is devised to minimize the HESS life cycle cost (LCC). Then, a dynamic traffic model composed of a Copula-based passenger flow generation method and a real-time timetable rescheduling algorithm incorporating a traction energy-passenger-time sensitivity matrix is developed to characterize multi-train traction load uncertainty. Furthermore, an MTRL algorithm based on a dueling double deep $Q$ network with knowledge transfer is proposed to simultaneously learn a generalized control policy from annealing task-specific agents and operation environments for solving the MTMDP effectively. Comparative studies based on a real-world subway have validated the effectiveness of the proposed framework for LCC reduction of HESS operation under urban rail traffic.