Real-Time Deep Reinforcement Learning Policy Controller for Magnetically Levitated Robots

Abstract

Magnetic levitation (Maglev) technology is widely adopted for transportation, photolithography, clean room, and material-handling applications, owing to its vibration- and friction-free, untethered motion. The magnetic movers are actuated using the stator coil array, which functions as mobile maglev robots (MLRs) with precision position and orientation motion control. Therefore, this article studies the MLRs for intelligent manufacturing, such as material handling and collaboration with manufacturing components, by integrating them with deep reinforcement learning (DRL) policy controllers. Linear DRL policies are developed and compared between Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO) for the horizontal motion control, which takes $\mathrm {455~\mu \text {s} }$ to compute the actuating forces for real-time operations. During policy training, random magnitude horizontal step responses are ticked every 500 ms to explore and exploit the dynamic capacity of the MLRs. The PPO policies are finalized based on the discovered deterministic nature of the maglev environment. Furthermore, the symmetry of the MLR configuration is utilized for policy training, with a threshold boundary included to enable MLRs to translate smoothly across the horizontal operating range. The trained policies exhibit aggressive oscillations and persistent offsets relative to References, respectively. To alleviate the fitting problem, distance-based soft state machines are introduced to integrate the DRL policy with the basic controller. The state machine parameters are empirically tuned using various location step responses. This feature enables the dynamic responses of the DRL policy to be combined with the precision responses at references. The soft state machines are also trained as auxiliary policy actions and evaluated using full operating-range trajectory tracking and long-stroke step responses alongside the policy and the basic controller. The DRL policy controllers outperform the basic controllers, with improved rise and settling times in step responses, while showing enhanced performance in tracking trajectories. Finally, the policy robustness is empirically evaluated. Note to Practitioners—This article is first motivated by the need for dynamic responses in long-stroke magnetically levitated (maglev) robots in manufacturing operations. The reported basic controllers are fine-tuned locally and implemented in the large horizontal operating space. Utilizing the patterns of the maglev robots (MLRs), a controller that provides dynamic responses based on the operating conditions is desired. Then, a secondary motivation for deep reinforcement learning (DRL) in smart manufacturing systems (SMSs) is investigated to obtain a policy controller using artificial intelligence. In addition to dynamic performance, achieving precise control with a DRL policy is challenging owing to the policy-fitting problem during training, which is unpredictable across different operating conditions. This study combines a basic controller and a DRL controller to switch between dynamic and precise operations smoothly. The proposed smart controllers are verified using an MLR, including long-trajectory tracking and multiple-step responses. Slight deviations between the MLR and the virtual environment are observed, and they remain confined to real-time control. The MLRs benefit SMSs as material-handling components, for swarm formation in production, and as manufacturing collaborators at the workstations. Future research will employ DRL and an improved training environment for motion and multi-MLR planners in swarm and collaborative applications.