Explorative Policy Optimization for industrial-scale operation of complex process control systems

Abstract

With the advancement of industrial automation, traditional process control methods increasingly struggle to manage the complex operational demands of industrial-scale chemical processes, particularly in the presence of unmodelled dynamics and high nonlinearity. This paper introduces an advanced reinforcement learning algorithm, Explorative Policy Optimization (EPO), specifically developed to optimize operational strategies, focusing on improving both production yield and product quality in such environments. The core innovation of the EPO algorithm is its exploration network, which dynamically adjusts exploration strategies based on discrepancies between predicted and actual values of state–action pairs, enabling more effective exploration. This approach improves decision-making by providing more accurate outcome assessments in complex and unmodelled conditions. EPO also integrates exploration data into the advantage function, ensuring a balance between exploration and exploitation, which is essential for optimizing performance in dynamic environments that require both safety and adaptability. EPO focuses on global optimization in processes with multiple operating conditions and steady states. It surpasses existing RL methods in overall performance while maintaining acceptable computational costs across a wide range of industrial settings. Its effectiveness and practicality are demonstrated through industrial-scale simulation experiments.