The end-to-end real-time scheduling method for hybrid flow shop based on heterogeneous cooperative multi-agent deep reinforcement learning

Abstract

Personalized customization increases the complexity of the hybrid flow shop production environment, which is characterized by the presence of variable discrete events. The buffer capacities of machines and the transportation times between processing stages and the warehouse influence the overall production process. Therefore, the research problem addressed in this paper is the real-time scheduling problem for hybrid flow shops with limited buffers and transportation time. To effectively tackle this challenging scheduling problem, this paper proposes a discrete-event-driven heterogeneous cooperative multi-agent deep reinforcement learning (HC-MADRL) framework. Within this framework, two types of heterogeneous scheduling agents are designed: the Job Agent (JA) and Stage-Machine Agents (SMAs). The JA employs an integrated LSTM-Pointer Network (LPN) architecture to select the most appropriate job based on the current job sequence states in real time. In contrast, the SMAs utilize a Transformer-based weighted network (TWN) incorporating a machine mask mechanism to assign the selected job to the optimal machine according to the state of its processing stage. Then, eight real-time job states and five generic stage-machine states are designed for the JA and SMAs, respectively. Meanwhile, a multi-level reward mechanism with reward shaping is proposed to guide the agents in learning to optimize the scheduling objective. Based on this, this paper presents an independent multi-agent proximal policy optimization (I-MAPPO) training algorithm that integrates two critic networks for heterogeneous multi-agent systems. Lastly, the experimental results demonstrate that the proposed method exhibits superior scheduling performance and real-time responsiveness compared with four benchmark real-time scheduling methods: PDRs, DQN, PPO, MAPPO and other two advanced multi-agent schedulers.