Monocular Reconstruction of Non-rigid Shapes Using Optical Flow Feedback

Abstract

In this paper we describe a variational approach to reconstruct the non-rigid shape from a monocular video sequence based on optical flow feedback. To obtain the dense 2D correspondences from the image sequence, which is critical for 3D reconstruction, we formulate the multi-frame optical flow problem as a global energy minimization process using subspace constraints, settles the problems of large displacements and high cost caused by dimensionality elegantly. Using the long-term trajectory tracked by optical flow field as input, our method estimate the depth of traced pixel in each frame based on the Non-Rigid Structure from Motion(SFM) algorithm. And finally, we refine the 3D shape via interpolation on recovered 3D point cloud and camera parameters. The experiment on real sequence of different objects demonstrates the accuracy and robustness of our framework.