Effective Deep Reinforcement Learning for Dynamic Machine Allocation: A Case Study on Metal Sputtering Tools

Abstract

Dynamic Machine Allocation (DMA) is a vital aspect of production scheduling in semiconductor manufacturing. Current DMA practices heavily rely on engineers’ domain expertise and require a few days of manual adjustments in response to rapid but significant fab changes, for example, due to unfamiliar economic shifts. Slow and heuristic DMA policy adaptation very often leads to production shortfalls. To reduce dependence on human expertise and speed up quality responses to changes, we design a framework of effective deep reinforcement learning (DRL) for DMA. Design innovations of the framework include (1) a discrete-event simulator for predicting production flows among machines with state, DMA action and reward aligned to fab practices; (2) a DRL neural network output transformation module that ensures action feasibility in task compatibility and machine availability; and (3) a DRL-based, two-stage agent of DMA policy learning that integrates DRL with optimization techniques for both efficient computation and quality DMA. Operation simulation by using the DMA case and data of a metal sputtering machine group demonstrates that our DRL-based design effectively learns DMA policies in different scenarios, each within one hour. In throughput performance, learned policies surpass a traditional heuristic by 3% to 20%. Our framework and the DRL-based method designs are generic and applicable to DMA of various machine groups.