Two-Timescale Volt/var Control Based on Reinforcement Learning with Hybrid Action Space for Distribution Networks

In volt/var control (VVC) for active distribution networks, it is essential to integrate traditional voltage regulation devices with modern smart photovoltaic inverters to prevent voltage violations. However, model-based multi-device VVC methods rely on accurate system models for decision-making, which can be challenging due to the extensive modeling workload. To tackle the complexities of multi-device cooperation in VVC, this paper proposes a two-timescale VVC method based on reinforcement learning with hybrid action space, termed the hybrid action representation twin delayed deep deterministic policy gradient (HAR-TD3) method. This method simultaneously manages traditional discrete voltage regulation devices, which operate on a slower timescale, and smart continuous voltage regulation devices, which function on a faster timescale. To enable effective collaboration between the different action spaces of these devices, we propose a variational auto-encoder based hybrid action reconstruction network. This network captures the interdependencies of hybrid actions by embedding both discrete and continuous actions into the latent representation space and subsequently decoding them for action reconstruction. The proposed method is validated on IEEE 33-bus, 69-bus, and 123-bus distribution networks. Numerical results indicate that the proposed method successfully coordinates discrete and continuous voltage regulation devices, achieving fewer voltage violations compared with state-of-the-art reinforcement learning methods.