WAYLA - Generating Images from Eye Movements

Abstract

We present a method for reconstructing images viewed by observers based only on their eye movements. By exploring the relationships between gaze patterns and image stimuli, the What Are You Looking At?" (WAYLA) system has the goal of synthesizing photo-realistic images that are similar to the original pictures being viewed. The WAYLA approach is based on the Conditional Generative Adversarial Network (Conditional GAN) image-to-image translation technique of Isola et al. We consider two specific applications - the first of reconstructing newspaper images from gaze heat maps and the second of detailed reconstruction of images containing only text. The newspaper image reconstruction process is divided into two image-to-image translation operations the first mapping gaze heat maps into image segmentations and the second mapping the generated segmentation into a newspaper image. We validate the performance of our approach using various evaluation metrics along with human visual inspection. All results confirm the ability of our network to perform image generation tasks using eye tracking data