Robust reinforcement learning for nonlinear process control with stability guarantees

Abstract

Reinforcement learning (RL) offers a promising route to fast, nonlinear feedback control for complex process systems; however, its deployment is hindered by the lack of formal stability guarantees and sensitivity to model-plant mismatch under constraints. This paper proposes a stability and robustness-oriented RL-based control framework for nonlinear constrained processes by explicitly integrating Lyapunov-based decision rules into the RL closed loop and importing an offset-free approach from model predictive control (MPC). The RL policy is treated as a performance-seeking candidate controller that is supervised by a Lyapunov-certified fallback controller: at each sampling instant, the learned candidate input is evaluated against a Lyapunov condition, and any violating proposal is rejected in favor of a stabilizing backup input, yielding Lyapunov-certified practical stability under sample-and-hold implementation. To mitigate steady-state offsets and enhance robustness to disturbances and mismatches, the state available to the learning agent is augmented with online-estimated uncertainty/disturbance variables in the spirit of offset-free MPC, enabling the policy/value function to condition its decisions on the magnitude of uncertainty rather than overfitting nominal dynamics. The proposed architecture is demonstrated using two representative RL methods—an HJB-based value-critic approach and a TD3-based actor-critic approach—both deployed under the same Lyapunov-supervisory switching logic. Simulation studies on nonlinear chemical process control problems show that the proposed RL-based control framework preserves the low online computational cost while enforcing Lyapunov stability and improving robustness under disturbances, thereby advancing RL toward reliable process control deployment.