Remote Guided Robotics via Leap Motion and Mixed Reality

The global teleoperation and telerobotics market is projected to reach $98.3billion (USD) by 2027. With the benefits of 5G coverage almost anywhere in the world will allow teleoperated robots to be implemented globally via the cloud. Remote Guided Robotics has excellent potential in existing markets, attempting to remove operators from dangerous tasks and providing access to labour in remote areas, where labour is scarce. This research introduces a configuration of technology that reduces the knowledge gap of what is required to successfully achieve high accuracy and realistic control of a robot in real-time. The system implemented in this paper utilises an infrared camera from Leap Motion to capture and track hand gestures and movements, a Kuka 6 DOF robot and software packages to follow the hand movements from the UltraLeap Leap Motion Controller (LMC) in real-time, and an HTC VR headset for a mixed reality experience where two cameras are placed in the robot work-cell to send back a 3D stereoscopic image to allow the operator to have an immersed experience when remotely controlling the robot. The system proved to have a repeatability of approx. $\pm \mathbf{10}\mathbf{mm}$, and an accuracy of 0.1mm. The gripper was able to successfully pick up 86% of the time, with a final test of gripping an item and guiding it through a bandsaw at an average rate of 24.2 seconds. Testings and simulations completed in this paper outline the level of accuracy and repeatability of the remote guide robotic system. This teleoperated robotic system has the ability to protect operators' safety and meet the explosive growth of needs in the various industries.