Deep Reinforcement Learning Based Multi-Timescale Volt/Var Control in Distribution Networks Considering Network Reconfiguration

Abstract

Coordinating Volt/Var control (VVC) across multiple timescales in distribution networks is challenging due to the diverse response characteristics of control devices. This paper proposes a novel bi-level data-driven multi-timescale VVC method to achieve coordinated control. The method integrates short-timescale control of continuous devices, such as photovoltaics, with longer-timescale control of discrete devices, including capacitor banks, and network reconfiguration. The VVC problem is formulated as a bi-level partially observable Markov decision process (POMDP). Inner-level control employs the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm for continuous devices, while outer-level control uses the Deep Double Q-Network (DDQN) algorithm for discrete devices and network reconfiguration. Collaborative training is achieved by aligning reward signals and providing inner-level agent actions as state information to outer-level agents. To mitigate over-exploration caused by network reconfiguration, graph neural networks (GNNs) are utilized to identify representative topologies, simplifying the reconfiguration space. The proposed method is validated on the IEEE 33-bus and PG&E 69-bus systems, demonstrating superior VVC performance and enhanced robustness to topological variations.