Online prediction-assisted safe reinforcement learning for electric vehicle charging station recommendation in dynamically coupled transportation-power systems

Abstract

With the proliferation of electric vehicles (EVs), the transportation network and power grid become increasingly interdependent and coupled via charging stations. The concomitant growth in charging demand has posed challenges for both networks, highlighting the importance of charging coordination. However, existing literature largely overlooks the interactions between power grid security and traffic efficiency, where the deterioration of grid security also leads to a decrease in traffic efficiency. In view of this, we study the en-route charging station (CS) recommendation problem for EVs in dynamically coupled transportation-power systems. The system-level objective is to maximize the overall traffic efficiency while enhancing the safety of the power grid. This problem is for the first time formulated as a constrained Markov decision process (CMDP), and an online prediction-assisted safe reinforcement learning (OP-SRL) method is proposed to learn the optimal and secure policy. To be specific, we mainly address two challenges. First, the constrained optimization problem is converted into an equivalent unconstrained optimization problem by applying the Lagrangian method, and then the Proximal Policy Optimization (PPO) method is extended to incorporate the constraint in the sequential decision process through the inclusions of cost critic and Lagrangian multiplier. Second, to account for the uncertain long-time delay between performing charging station recommendation and commencing charging, we put forward an online sequence-to-sequence (Seq2Seq) predictor for state augmentation, offering foresightful information to guide the agent in making forward-thinking decisions. Finally, we conduct comprehensive experimental studies based on the Nguyen-Dupuis network and a large-scale real-world road network, coupled with IEEE 33-bus and IEEE 69-bus distribution systems, respectively. Results demonstrate that the proposed method outperforms baselines in terms of road network efficiency, power grid safety, and EV user satisfaction. The case study on the real-world network also illustrates the applicability in the practical context.