Design of a human-robot interaction system for robot teleoperation based on digital twinning

Human-robot interaction (HRI) research using vision for robot teleoperation is closely related to the development of the field of artificial intelligence. There are many teleoperation methods for imitating human postures by humanoid robots. However, it is not easy to fully parameterize human movements using collaborative robots because of the difference in morphology between humans and collaborative robots. This paper proposes a robot teleoperation and human-robot interaction method based on digital twin technology to try to solve this problem. Using this method, the operator can control the robot at a distance to accomplish complex tasks in an unstructured environment. The system integrates virtual reality technology to provide the operator with a first-person view of the robot. Then, a 3D human pose detection algorithm is proposed, through which the 3D coordinates of human torso, shoulder, elbow, and hand are captured and a digital twin model of the human body is created. In addition, a geometric vector method is proposed to calculate the upper limb joints of the human body and convert them into motion commands of the robot. Finally, teleoperated human-robot interaction experiments were conducted by the ELITE robot platform. We show that the system has good action imitation and dynamic behavior adaptation, which proves the effectiveness and feasibility of our proposed method.