Model-based face tracking for dense motion field estimation

When estimating the dense motion field of a video sequence, if little is known or assumed about the content, a limited constraint approach such as optical flow must be used. Since optical flow algorithms generally use a small spatial area in the determination of each motion vector the resulting motion field can be noisy, particularly if the input video sequence is noisy. If the moving subject is known to be a face, then we may use that constraint to improve the motion field results. This paper describes a method for deriving dense motion field data using a face tracking approach. A face model is manually initialized to fit a face at the beginning of the input sequence. Then a Kalman filtering approach is used to track the face movements and successively fit the face model to the face in each frame. The 2D displacement vectors are calculated from the projection of the facial model, which is allowed to move in 3D space and may have a 3D shape. We have experimented with planar, cylindrical, and Candide face models. The resulting motion field is used in multiple frame restoration of a face in noisy video.