Data-driven hierarchical multi-policy deep reinforcement learning framework for multi-objective multiplicity dynamic flexible job shop scheduling

Abstract

In the context of Industry 4.0, manufacturers face pressure to personalize products and accelerate the supply chain. This requires rapid response to volatile production schedules, ensuring a balance between operational efficiency and product quality. Moreover, the rapid development and convergence of the cloud computing, Internet of Things (IoT), and big data have expanded the need for real-time tracking and adaptive scheduling to address uncertainties, such as equipment downtime, supply variation, and ongoing product revisions. The capability of IoT has significantly improved the continuous monitoring and data analysis, emphasizing the importance of developing effective real-time scheduling solutions in the manufacturing system. In response to these evolving industrial requirements, and driven by objectives to reduce the makespan, total tardiness, and energy consumption, we study the multi-objective multiplicity dynamic flexible job shop scheduling problem (MOMDFJSP), to cope with the challenges of new order arrivals and machine breakdowns in the IoT-enabled manufacturing system. This study proposes a novel hierarchical multi-policy deep reinforcement learning framework for IoT-infused manufacturing environments, aiming to integrate these diverse requirements and uncertainties into a coherent and responsive scheduling framework. The proposed framework comprises an upper-level control policy network and three lower-level objective policy networks. The upper-level and lower-level networks are respectively responsible for selecting temporary optimization objectives and specific dispatching rules. Based on the proposed framework, we design a two-stage training approach named the hierarchical multi-policy soft actor-critic (HMPSAC) algorithm to train multiple policy networks. In addition, we develop a fluid model to design the state features and dispatching rules that act as inputs and outputs, respectively, for the deep reinforcement learning (DRL) policy network. The comparative analysis with well-known dispatching rules and DRL-based methods reveals the superior performance of HMPSAC algorithm.