Eye Tracking and Emotion Recognition Using Multiple Spatial-Temporal Networks

Abstract

E-commerce products need to be measured by reader responses as a more objective evaluation. Some of them are through emotion expression identification or eye-tracking. Using these two variables from video capture provides a more thorough evaluation of the response to interest and emotion. This study proposes a spatial-temporal multi-networks method using Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) from video for 60 seconds. The results showed that two classes of emotional expression and four directions of eye-tracking gave better accuracy, namely 95.83%, compared to three classes of emotion and four directions of eye-tracking, which was 91.67%. Experiments also show that using CNN-LSTM significantly increased accuracy, while the weight correction technique does not have much effect. The evaluated F1 score shows the consistency of the proposed model.