A Novel Transformer-based Framework for Multi-View 3D Human Mesh Reconstruction

Abstract

This paper addresses two key problems of multi-view 3D Human Mesh Reconstruction (HMR): the difficulty of fusing features from multiple images and the lack of training data. We design a novel Transformer-based framework called Multi-View Human Mesh Transformer (MV-HMT), which is comprised of parallel Tiny CNNs and Transformer Encoder. MV-HMT takes multi-view silhouette as inputs, regresses the parameters of human shape and pose, and is effective for multi-view feature fusion. Real-Time Data Synthetic (RT-DS) technique is proposed in this work to solve the second problem. RT -DS is a plug-and-play component that generates paired silhouettes-mesh on CUDA, and provides an inexhaustible supply of synthesis data for pre-training of the neural network. Our method outperforms existing methods for multi-view HMR on the four-view datasets MPI-INF-3DHP and Human3.6M. Another new three-view dataset, MoVi, with more subjects and more accurate annotation, was used to evaluate the generality of our method and showed remarkable results.