Highly Accurate and Fully Automatic Head Pose Estimation from a Low Quality Consumer-Level RGB-D Sensor

In this paper we describe a novel algorithm for head pose estimation from low-quality RGB-D data acquired using a consumer-level device such as Microsoft Kinect. We focus our attention on the well-known challenges in the processing of depth point-clouds which include spurious data, noise, and missing data caused by occlusion. Our algorithm performs pose estimation by fitting a 3D morphable model which explicitly includes pose parameters. Several important novelties are described. (i) We propose a method for automatic removal of the majority of spurious depth data which uses facial feature detection in the associated RGB image. By back-projecting the corresponding image loci and intersecting them with the 3D point-cloud we construct the facial features plane used to crop the point-cloud. (ii) Both high convergence speed and high fitting accuracy are achieved by formulating the fitting objective function to include both point-to-point and point-to-plane point-cloud matching terms. (iii) The effect of misleading point-cloud matches caused by noisy or missing data is reduced by using the Tukey biweight function as a robust statistic and by employing a re-weighting scheme for different terms in the fitting objective function. (iv) Lastly, the proposed algorithm is evaluated on the standard benchmark Biwi Kinect Head Pose Database on which it is shown to outperform substantially the current state-of-the-art, achieving more than a 20-fold reduction in error estimates of all three Euler angles i.e. yaw, pitch, and roll. A thorough analysis of the results is used both to gain full insight into the behaviour of the described algorithm as well as to highlight important methodological issues which future authors should consider in the evaluation of pose estimation algorithms.