Emotion Recognition by Integrating Eye Movement Analysis and Facial Expression Model

Abstract

This paper presents an emotion recognition method which combines knowledge from the face and eye movements to improve the system accuracy. Our method has three fundamental stages to recognize the emotion. Firstly, we use a deep learning model to obtain the probability of a sample belonging to each emotion. Then, the eye movement features are extracted from an open-source framework which implements algorithms that demonstrated state-of-the-art results in this task. A new set of 51 features have been used to obtain related information about each emotion for the corresponding sample. Finally, the emotion for a sample is recognized based on the combination of the knowledge from the two previous stages. Experiment on the validation set of Acted Facial Expressions in the Wild (AFEW) dataset shows that the eye movements can make 2.87% improvement in the accuracy for the face model.