Intuitive hand motion-based teleoperation system for human-mobile manipulator interaction using mixed reality

This paper presents an intuitive teleoperation interface for human-mobile manipulator interaction, addressing the challenges posed by unstructured, dynamic environments and the inherent uncertainties in robot dynamics. Traditional teleoperation systems often lack intuitiveness and immersiveness, particularly when controlling high-DoF (degree-of-freedom) robotic systems. The proposed approach integrates IMU (Inertial Measurement Unit)-based hand motion tracking with mixed reality technology, enabling seamless control in both Manipulation and Locomotion modes. In Manipulation mode, a torque-based controller regulates the motion of the manipulator’s end-effector, while a radial basis function neural network compensates for dynamic uncertainties. The null-space projection method further provides self-autonomous redundancy resolution. In Locomotion mode, a velocity-based control strategy is implemented to govern the motion of the mobile platform. The mixed reality environment, constructed using SLAM (Simultaneous Localization and Mapping), 3D point-cloud data, and real-time 2D visual streaming, provides immersive and comprehensive visual feedback through a head-mounted display device. Stability of the Manipulation mode is analyzed via Lyapunov stability theorem and validated through numerical studies. Experiments conducted on a mobile manipulator, comprising a differential-drive base and a 7-DoF manipulator, demonstrate effective performance in complex scenarios. Additionally, comparative experiments with multiple input devices show that the proposed system offers superior usability and imposes a lower operator workload than alternative interfaces. These results highlight the robustness, intuitiveness, and practical applicability of the proposed method in dynamic and hazardous environments.