Model-free safe deep reinforcement learning for grid-to-vehicle management considering grid constraints and transformer thermal stress

Abstract

The increasing penetration of Electric Vehicles (EVs) presents challenges to the distribution grid, due to more volatile power profiles and higher peak demand. One key research question is how to accommodate EVs with limited-capacity grid equipment, such as transformers and lines. However, uncertainties from the EV side and the complexity of grid equipment models challenge the performance of the control strategies implemented. Moreover, the thermal loading of the transformer is often neglected. In this work, we propose a fully model-free, safe Deep Reinforcement Learning (DRL)- based grid-to-vehicle management strategy to avoid electric and thermal overloading of the transformer and power grid constraint violation. The management strategy is based on Projection-based Constraint Policy Optimization (PCPO) and takes only the observable information from the grid and vehicles. The target is to maximize energy delivery to the EV fleet while considering safe constraints, such as transformer thermal loading, voltage magnitude limits, and line loading limits. We compared the proposed strategy with conventional DRL and other safe DRL methods and investigated its robustness against higher ambient temperatures. The results show that the proposed strategy can deliver 92 % energy and reduce violations of the grid and transformers, while the other benchmarks deliver less than 80 %. The robustness test demonstrates that the proposed strategy is effective in various temperature. Moreover, the proposed strategy can effectively reduce at most 90 % of the transformer aging incurred by the thermal stress, compared with the uncontrolled charging.