Chance Constrained MDP Formulation and Bayesian Advantage Policy Optimization for Stochastic Dynamic Optimal Power Flow

Abstract

Although deep reinforcement learning based on Markov Decision Process (MDP) constitutes a well-suited method for real-time control under uncertainties, its application to stochastic dynamic optimal power flow (SDOPF) problem is still challenging in the presence of increasing proliferation of various distributed energy resources, driven by its limitations on constraints satisfaction under uncertainties. While pioneering research explored Constrained MDP and Risk-Aware MDP formulations of SDOPF pursuing cumulative constraint violation minimization, they both struggle with satisfaction of state-wise safety constraints. This letter proposes a Chance Constrained MDP formulation of SDOPF and a Bayesian advantage policy optimization solution method. Bayesian neural networks are used to construct the probability distributions of state- and trajectory-wise constraint violations, and a novel advantage function is incorporated to improve both the policy's quality and safety. Case studies validated the cost-efficiency and comprehensive safety performance of the proposed method against the state-of-the-art.