A hybrid framework of intrinsic constraint handling and safe reinforcement learning for crude oil scheduling

Abstract

Crude oil scheduling is a critical but highly complex sequential decision-making problem in refinery operations. Traditional mathematical programming methods suffer from exponential computational complexity with increasing scale, while traditional reinforcement learning approaches struggle to guarantee the satisfaction of numerous process and product quality constraints. This highlights a critical issue in the current field of scheduling: the lack of an optimization methodology that can simultaneously achieve high computational efficiency and robust constraint satisfaction. To bridge this gap, we propose a novel hybrid framework based on constraint stratification. The framework embeds critical hard constraints, such as entity connectivity limit, directly into the scheduling environment for intrinsic satisfaction through action masking and shaping. Concurrently, it employs safe reinforcement learning algorithms to manage soft constraints, such as tank inventory levels and product quality specifications, by optimizing a primary objective while keeping constraint violations below a predefined threshold. Through comparative experiments on scheduling cases, the Constrained Policy Optimization algorithm was identified as the most effective safe reinforcement learning method. The results demonstrate that our proposed framework significantly outperforms traditional methods. It achieves the high computational efficiency and scalability of reinforcement learning while providing a much stronger safety guarantee than penalty-based approaches, offering a robust and practical solution for complex industrial scheduling problems.