Pose driven motion image generation aided by depth information

Motion transfer which can be used as drive technology of the interaction between users and virtual roles has been a research hotspot in recent years. It is essentially a deformation process of human appearances, consequently motion transfer is typically regarded as a pose-guided image generation task which can be solved by a GAN based framework. However, the real motions occur in 3D space and the image generation in 2D plane inevitably lacks the depth information guidance of the original motions, which will result in the confusion of depths for different body parts. Besides, the adversarial loss of GAN presents weak constraints on the silhouette details. In this paper, we propose a two-stage GAN based model to make up for the defect of lacking depth information and improve the accuracy of the generated silhouette details. In stage-I, we propose a silhouette attention GAN with a silhouette consistency loss to generate the depth maps of target people. This not only brings the depth information of the original motions in 3D space but also aligns the body regions with reliable silhouettes for the following person image generation. In stage-II, we propose a context-enhanced GAN with the target poses and depth maps generated in the first stage as input to generate the final motion images. The generated results have reliable depth information and accurate silhouettes, demonstrating the effectiveness of the proposed model.