Multi-modal object of interest detection using eye gaze and RGB-D cameras

Abstract

This paper presents a low-cost, wearable headset for mobile 3D Point of Gaze (PoG) estimation in assistive applications. The device consists of an eye tracking camera and forward facing RGB-D scene camera which are able to provide an estimate of the user gaze vector and its intersection with a 3D point in space. A computational approach that considers object 3D information and visual appearance together with the visual gaze interactions of the user is also given to demonstrate the utility of the device. The resulting system is able to identify, in real-time, known objects within a scene that intersect with the user gaze vector.