A deep reinforcement learning approach with graph attention network and multi-signal differential reward for dynamic hybrid flow shop scheduling problem

Abstract

In real-life manufacturing systems, production management often faces uncertainty due to urgent demands and dynamic job insertions. Such uncertain environments pose significant challenges for scheduling, particularly in minimizing delivery delays and improving overall efficiency. Deep reinforcement learning (DRL) brings potential for rapid real-time production decisions, but scheduling in these environments with the objective of reducing delivery delays remains a challenging problem. This paper investigates a hybrid flow-shop dynamic scheduling problem with job insertions for minimizing the total weighted tardiness (TWT). An end-to-end DRL based method, the proximal policy optimization with graph attention network (PPO-GAT), is proposed to address the problem. First, a multi-agent system is established to simulate the actual manufacturing system and serve as a foundation for implementing intelligent production scheduling. Then, a novel graph-based state representation is developed to observe instantaneous states for the hybrid flow-shop. Two graph models are designed to represent system features and job features, and are extracted and fused by graph attention networks (GAT) to form the global feature. Afterwards, a multi-signal differential reward (MSDR) function is designed to address the intractable reward sparsity caused by the TWT objective. Finally, ablation experiments are conducted to validate all the proposed algorithmic components, and the PPO-GAT is compared with benchmark methods. Experimental results demonstrate the superiority of the proposed GAT, MSDR, and PPO-GAT. Moreover, the PPO-GAT has been proven to make real-time scheduling decisions for hybrid flow-shops with any scale, which can be considered as a promising solution for extensive industrial applications.