Virtual Character Animation based on Data-driven Motion Capture using Deep Learning Technique

Perceptions in motion capture (mocap) technology are increasing every day as the variety of applications using it is doubling. By leveraging the resources offered by mocap technology, human activity characteristics are captured and can be used as the source for animation. The devices involved in the technology are therefore very costly and hence not practical for personal use. In this scenario, we implement a framework capable of producing mocap data from standard RGB video and use it to animate a character in 3D space, based on the action of person in the original video with the help of deep learning techniques. The Human Mesh Recovery (HMR) scheme is used to extract mocap data from the input video to determine where joints of the person in the input video are located in 3D space, using 2D pose estimation. The locations of 3D joints are used as mocap data and transferred to Blender with a simple 3D character using which the character can be animated. A subjective evaluation of our framework based on the metric called observation factor was performed and yielded an accuracy value of 73.5%.