Curriculum Multi-Stage Reinforcement Learning for Automated Interlinked Production Systems on Virtual Commissioning Simulations

In order to automate the software engineering process of interlinked production systems, reinforcement learning applications can be used to learn the control flow logic on the basis of virtual production systems. Since the simulation- based prototypes are available for virtual commissioning (VC) anyway, they can be used simultaneously as reinforcement learning environments. In this work, the event-discrete flow logic for the transport and assembly of a target workpiece is learned automatically by reinforcement learning on the real use case of the VC simulation of a PLC-based production system. According to the idea of curriculum learning, the system is trained separately in subsystems to support its modularity and to reduce the complexity of the overall learning process. With regard to the learning processes, subsystems, sequence errors, termination criteria and necessary action and state adjustments typical for the PLC-based plant are identified and implemented in the VC simulation. The reward functions are derived with respect to the individual subsystems. The learned controls of the subsystems are then merged back together for a complete flow of the entire system.