Comparison of reinforcement learning techniques for controlling a CSTR process

Abstract

One of the main promises of Industry 4.0 is the incorporation of computational intelligence techniques in industrial process control. For the chemical industry, the efficiency of the control strategy can reduce the production of effluents and the consumption of raw materials and energy. A possible, although currently underutilized approach is reinforcement learning (RL), which can be used to optimize many sequential decision making processes through training. This work used Van de Vusse kinetics as an evaluation environment for controllers based on reinforcement learning and comparison with conventional solutions like non-linear model predictive control (NMPC). These kinetics contain characteristics that make it difficult for classic controllers such as PID to handle, such as its non-linearity and inversion point. The investigated algorithms showed excellent results for this notable chemical process control benchmark. This study was divided into two experiments: setpoint change and operation around the inversion point. The former showed the ability of RL controllers to adjust the controlled variable and simultaneously maximize production. The latter revealed the excellent management capability of the reinforcement learning algorithms and NMPC at the inversion point. In this study, the RL algorithms performed similar to NMPC but with lower computational cost after training.