Deep reinforcement learning driven by heuristics with Petri nets for enhancing real-time scheduling in robotic job shops

In robotic job shops (RJS), a significant challenge lies in optimizing task allocation and robot routing simultaneously, especially since these tasks must be accomplished in real-time to efficiently manage unexpected situations, such as the urgent need for AGV recharging or sudden order additions. Deep reinforcement learning (DRL) shows promise for these complex scheduling tasks due to its ability to address problems characterized by substantial computational complexity. However, the rapid expansion of RJS state space and the difficulty of avoiding cyclic loops for AGVs pose significant challenges for DRL in realistic settings. To address these, we present a novel approach combining an artificial-potential-field (APF) with a deep Q-network (DQN) in a Petri net framework. The APF is designed for Petri nets to guide token movement toward goal place nodes. Throughout the learning process, the APF-guided mixed policy employs a cosine-annealing probability for APF policy and a piecewise linear probability for random policy. Initially, action selections predominantly rely on APF policy to efficiently gather high-reward experience. As training progresses, they shifts to more rely on the learned neural-network policy, with random exploration supplementing diversity, ensuring a robust transition from reward-driven exploration to precise decision-making. The APF-DQN method is tested in real-world RJS scenarios, showing superior exploration success and training efficiency over baseline DQN. It significantly outperforms both conventional dispatching rules and baseline DQN, reducing average makespan by over 55% compared to dispatching rules and by 14.9% relative to baseline DQN. This method significantly enhances traditional DQN by improving exploration success, learning efficiency, policy convergence, and adaptability to dynamic environments.