Modified deep reinforcement learning for frequency regulation in active distribution systems with soft open points, storage units and electric vehicles

As an effective approach to achieving the smart grid concept and reducing carbon emissions, the integration of renewable energy sources and storage devices is increasing. However, with the growing demand for high-power charging, the electrical grid faces significant challenges as electric vehicle (EV) adoption rises, particularly in the presence of stochastic energy sources. Consequently, the need for robust regulation strategies to manage distribution system uncertainties, especially in frequency regulation, is becoming more critical. Distribution systems, interconnected through multi-terminal soft open points (SOPs), are evolving into highly controllable, integrated, and flexible architectures. Performance is further enhanced by incorporating a dedicated terminal for hybrid hydrogen energy storage. Additionally, the integration of vehicle-to-grid (V2G) and grid-to-vehicle (G2V) operations has been explored. To effectively manage these operational frameworks, a modified deep reinforcement learning (RL) strategy based on the deep deterministic policy gradient (DDPG) algorithm is proposed. Multi-agent deep RL is employed, generating multiple control signals per agent based on reward functions derived from a quadratic optimization function within the model predictive control (MPC) framework. To ensure an optimal control action waveform and enhance system performance, each DDPG deep RL agent's control action value is scaled by its observation value, integral, and derivative, integrated through a filter element. When applying the proposed modified deep RL strategy alongside the components, the rate of frequency change and power transfer fluctuations achieved minimal steady-state errors in the range of × 10⁻⁸, with significantly damped overshoot and undershoot levels. This approach effectively maintains system performance, outperforming other simulated scenarios.