Toward Universal Controller: Performance-Aware Self-Optimizing Reinforcement Learning for Discrete-Time Systems With Uncontrollable Factors

Abstract

The industrial system usually contains not only controllable variables (CVs) but also uncontrollable variables (unCVs), e.g., weather conditions and friction. These unCVs have a direct impact on system control performance. Despite the success of current deep reinforcement learning (DRL) control algorithms, most of them neglect the impact of unCVs, which can cause the deterioration of control performance and instability of the system. To perceive and eliminate the impact of unCVs, a performance-aware self-optimizing universal controller (PASOUC) is designed in this article. The PASOUC aims at integrating the representation of unCVs and controller design to perceive and eliminate the impact of unCVs under different conditions, which goes beyond most existing control methods. Technically, a historical trajectory-inspired control performance perceptron is developed to perceive the impact of unCVs on system control performance under different conditions. Subsequently, a new performance-aware reward is designed to integrate the representation of unCVs and controller design while training the DRL controller. In addition, the domain randomization (DR) training strategy is employed to learn a universal control policy, which can access the approximate optimal trajectory under nonideal conditions. In this way, the impact of unCVs can be eliminated. To handle the low efficiency of the DR training, the policy improvement-policy proximal optimization (PI-PPO) is proposed to enhance the convergence speed of the DR training by performing explicit policy improvement. Finally, illustrative examples are presented to demonstrate the superiority of the proposed method.