Two-Level Pareto-Based Hierarchical Reinforcement Learning for Dynamic Multi-Objective Scheduling in Semiconductor Fabs

Abstract

This paper addresses the critical challenges of dynamic, multi-objective scheduling in semiconductor wafer manufacturing, where decision makers must balance conflicting goals such as throughput, yield, cycle time, and energy efficiency amid frequent disruptions and large-scale complexity. We propose a novel Pareto-based Hierarchical Multi-Objective Reinforcement Learning (PH-MORL) framework, which integrates the exploration of Pareto-optimal trade-offs with a hierarchical decision structure. The high-level agent adaptively determines global objective pReferences by referencing a continuously maintained Pareto archive, capturing a diverse set of non-dominated solutions. These preferences are then broadcast to low-level agents that learn fine-grained scheduling actions for individual workstations. Through this dual-layer synergy, PH-MORL overcomes the limitations of fixed-weight multi-objective methods and single-layer learning architectures, enabling real-time reconfiguration of scheduling strategies in response to machine failures, urgent orders, or yield fluctuations. The proposed framework not only discovers a broad spectrum of feasible performance compromises but also dynamically coordinates local and global decision making. Our formal analysis establishes tracking and stability properties under standard finite-state stochastic-approximation and contraction assumptions, while extensive discrete-event simulation experiments in realistic wafer-fab settings demonstrate improved dynamic scheduling trade-offs, real-time inference feasibility, and competitive performance against rolling-horizon NSGA-II/III baselines. We further provide CPU-only end-to-end inference measurements that include state-feature assembly, feasible-action enumeration and masking, neural-network inference, rule-based safety filtering, and dispatch commitment. Across 30 independent 240-hour replications, PH-MORL achieves 15.7 ms mean latency, 31.5 ms p95 latency, 50.3 ms p99 latency, and 182.0 ms maximum latency, remaining well below the 1 s dispatching budget required for real-time dispatching in modern 300 mm fabs. These results highlight PH-MORL’s potential for scalable and flexible optimization in modern semiconductor manufacturing, and its generalizable structure paves the way for broader adoption in other complex, multi-objective industrial environments.