Entropy-maximizing TD3-based reinforcement learning for adaptive PID control of dynamical systems

Abstract

The proper tuning of proportional–integral–derivative (PID) control is critical for satisfactory control performance. However, existing tuning methods are often time-consuming and require system models that are difficult to obtain for complex processes. To this end, automatic PID tuning, particularly that based on deep reinforcement learning, eliminates the necessity of a system model by treating the PID tuning as a black-box optimization. However, these methods suffer from low sample efficiency. In this paper, we present an entropy-maximizing twin-delayed deep deterministic policy gradient (EMTD3) method for automatic PID tuning. In our method, an entropy-maximizing stochastic actor is deployed at the beginning to ensure sufficient explorations, followed by a deterministic actor to focus on local exploitation. Such a hybrid approach can enhance the sample efficiency to facilitate the PID tuning. Extensive simulation studies are provided to show the superior performance of the proposed method relative to other methods on data efficiency, adaptivity, and robustness.