An offline-to-online reinforcement learning framework with trajectory-guided exploration for industrial process control

Abstract

Reinforcement learning (RL) in industrial process control faces critical challenges, including limited data availability, unsafe exploration, and the high cost of high-fidelity simulators. These issues limit the practical adoption of RL in process control systems. To address these limitations, this paper presents a comprehensive framework that combines offline pre-training with online finetuning. Specifically, the framework first employs offline RL method to learn conservative policies from historical data, preventing overestimation of unseen actions. It then transitions to fine-tuning using online RL method with a mixed replay buffer that gradually shifts from offline to online data. To further enhance safety during online exploration, this work introduces a trajectory-guided strategy that leverages timestamped sub-optimal expert demonstrations. Rather than replacing agent actions entirely, the proposed method computes a weighted combination of agent and expert actions based on a decaying intervention rate. Both components are designed as modular additions that can be integrated into existing actor-critic algorithms without structural modifications. Case studies on penicillin fermentation and simulated moving bed (SMB) processes demonstrate that the proposed framework outperforms baseline algorithms in terms of learning efficiency, stability, computation costs, and operational safety.