Reinforcement Learning for Stability-Guaranteed Adaptive Optimal Primary Frequency Control of Power Systems Using Partially Monotonic Neural Networks

Abstract

Deepening the deployment of distributed energy resources requires the large-scale integration of inverter-based resources, which can deteriorate the frequency stability. Recent studies propose using neural Lyapunov-based reinforcement learning for control. While this method can be trained offline with performance guarantees, it is only optimal for specific values of system parameters, as it omits critical modeling factors like decreasing inertia and damping variation over time. To maintain the performance at varying operation points, we consider an adaptive neural Lyapunov framework that adapts the controller’s output in the presence of varying parameters. Neural networks require flexibility to maximize adaptive control performance, while stability demands monotonicity, creating an inherent conflict. In this paper, we design a partially monotonic controller that maintains stability with maximal representation capacity for parameter adaptation. Stability is ensured by having monotonicity retained for frequency while non-monotonicity is allowed for the system parameters, such as damping and inertia. The structural form of partially monotonic neural networks is used for the controller design to that end. Flexibility is allowed by the design when adaptation to changes to the system parameters is made, while the Lyapunov stability guarantee is retained. The non-monotonic layers are chosen through an adaptive layer that is designed for damping and inertia based on their relationship to control in the system equation, by which optimized output for different operating conditions is allowed.