VR Eye-Tracking using Deflectometry

We present a novel approach for accurate eye tracking as required, e.g., in VR/AR/MR headsets. Our method exploits the retrieved surface normals and dense 3D features extracted from deflectometry measurements to estimate the gazing direction. © 2021 The Author(s) 1.​ ​Introduction Although the task has been studied for several decades, a robust solution to accurate and fast eye tracking remains an unsolved problem. With the advent of Virtual, Augmented, or Mixed Reality (VR/AR/MR), accurate eye tracking recently attracted considerable research interest mainly because it enables many functions that significantly improve the performance and experience of VR/AR/MR headsets, such as foveated rendering, or compensating for the accommodation-convergence reflex. To estimate the gazing direction of the human eye, current approaches either utilize 2D features detected from 2D eye images, or exploit sparse reflections of a few point light sources at the eye surface (“corneal/scleral reflections”). The latter retrieves 3D surface information for an improved gazing direction calculation, albeit only at maximal ~10 surface points. In this contribution we introduce an approach that significantly increases the information content provided from corneal or scleral reflections by using Deflectometry to acquire a dense and precise 3D model of the eye surface. The acquisition of ~1 million surface points per measurement step is easily achievable with off-the-shelf hardware. We exploit the retrieved surface normals and dense 3D features estimated via deflectometry to accurately estimate the gazing direction. 2. Method and Results Deflectometry is an established method in surface metrology to reconstruct the 3D surface of specular objects, such as freeform lenses, car windshields, or technical parts [1-3]: The reflection of a screen displaying a known pattern (e.g. a sinusoid) is observed after reflection from the specular surface under test. From the deformation of the pattern in the camera image, the normal vectors of the surface (and eventually the surface shape via integration) can be calculated. The inherent depth-normal-ambiguity is solved by adding a second camera, which results in a so-called “Stereo-Deflectometry” system [1]. Our proposed method utilizes Deflectometry for a dense and precise measurement of the eye surface. To calculate the gazing direction we first trace back the measured surface normal vectors towards the center of the eye. Due to the vastly different radii of cornea and sclera, the back-traced surface normals aggregate at two points inside the virtual 3D eye model: the center of the corneal sphere and the center of the scleral sphere (see Fig.1.c). Eventually, we calculate the Fig. 1. Calculating the gazing direction using dense 3D surface measurements. a) Deflectometry measurement: Camera image of the sinusoidal screen pattern reflected from the eye surface. b) Error map: calculated normal map w.r.t. the ground truth (error in degrees). c) Calculation of the gazing direction by tracing back the measured surface normals to the scleral and corneal center. CF2E.3.pdf OSA Imaging and Applied Optics Congress 2021 © OSA 2021