Dynamic Surface Capture for Human Performance by Fusion of Silhouette and Multi-view Stereo

We present a multi-camera based 3D dynamic surface capture solution, which supports high-fidelity generation of 3D dynamic content for various performance scenes. Our system uses a set of RGB cameras to synchronously acquire scene image sequences and performs a processing pipeline to produce 4D videos. We propose a multi-view point cloud reconstruction method which integrates volumetric based guidence and contraint into a coarse-to-fine depth estimation framework. It gives accurate point cloud models and can handle well with the scenes where textureless objects and multiple subjects present. We also present new methods and introduce modifications for several other key computation modules of the processing pipeline, including foreground segmentation, multi-camera calibration, mesh surface reconstruction and registration, texturing, 4D video compression, etc. Experiments on captured scenes show that our system can produce highly accurate and realistic 4D models of the human performances. We develop a 4D video player and toolkit plugins, and demonstrate the use of integrating the 4D content in VR and AR applications.