Human Motion Retargeting to a Full-scale Humanoid Robot Using a Monocular Camera and Human Pose Estimation

The retargeting human motions to those of a humanoid robot is a difficult task that involves using complex humanoid models and intensive geometric calculations, while also requiring high joint recognition accuracy. Herein, we propose a new motion retargeting framework for whole-body motions using a monocular camera composed of three modules per frame: 1) the extraction of 3D human joint coordinates from a package from pose AI, 2) the calculation of human joint angles by fitting the humanoid model to the skeleton model attained from the pose AI using a global optimization method, and 3) the transmission of the estimated joint angles as a pose command to a full-scale humanoid robot. The results suggest that the proposed framework can reproduce human-like motion sequences while reflecting certain limitations in the robot’s joint angles due to intentionally set hardware limitations and constraints. Using the proposed method, the robot can directly mimic human motion at the joint angle level based solely on images taken by an RGB camera or video files. These findings suggest that it would be useful to construct big data consisting of joint angle vectors for various human poses and joint angle trajectories for human motions, so that—as one example application in the near future—a robot butler could refer to these big data when performing various motions at a person’s home or in the office.