Synchronized Control of Robotic Arm based on Virtual Reality and IMU Sensing

This study introduces a robotic control system that combines virtual reality integration and inertial measurement units (IMU) using mixed reality (MR) devices. The system integrates three main modules: (1) virtual reality (VR), which simulates remote reality by the Hololens2, (2) the wearable IMUs device, which captures the operator's hand movements; and (3) the robotic arm UR5, which is controlled by the user and the VR environment. The virtual and physical systems communicate via a Message Queuing Telemetry Transport (MQQT) communication architecture to establish communication between modules. To introduce a closed-loop control system for the robotic arm, model predictive control (MPC) was achieved with precise path planning to provide a flexible, intuitive, and reliable method to operate the remote-controlled manipulator. To validate the system integration and functions, two demonstrations were conducted: (a) the offline mode, where the VR module of the robotic arm was controlled by the IMUs device to check correctness, and (b) the online mode, where the control command was transferred to UR5 to complete a target mission via artificial potential field (APF) adjustment. The primary outcome of this study was the development of virtual and real industrial robotic arms to test the developed model in VR and shop floor labs.