Optimizing Weights to Fit Parametric Operation Policies for Generalized Working Conditions in Linear Systems Using Deep Reinforcement Learning

Abstract

At present, working conditions are becoming more complex, and operation policy requirements are more diverse in process system engineering. To control a process problem, a balance must be found between speed and stability, in that operations should sometimes be faster and other times smoother. Traditional controllers, such as PID and model predictive control are applied in various problems, and some parameters in controllers can be used to represent the operation policy. However, there can be difficulties in tuning parameters, and time costs of online calculation. This article proposes parametric deep reinforcement learning (PDRL) to replace traditional controllers. PDRL has two parts. A vanilla DRL framework is adapted to solve the setpoint tracking problem. With a state and a reward function and robust training tricks, trained agents can be applied to more generalized working conditions. Base agents of different operation policies are trained in advance. With target performance from operators, the target policy can be fitted by base agents with a set of weights, which are first optimized by minimizing the squared error between the target and fitted policy in a basic task, and applied to generalized conditions. A shell benchmark problem is chosen as a case study, whose results show that PDRL has feasibility and stability both in basic and generalized tasks, even in a noisy environment.