Emotion recognition in Virtual Reality using sensor fusion with eye tracking

Abstract

Emotion recognition is an emerging field with applications in healthcare, education, and entertainment. This study integrates Virtual Reality (VR) with multi-sensor fusion to enhance emotion recognition. The research comprises two phases: data collection and analysis/evaluation. Ninety-five participants were exposed to curated audiovisual stimuli designed to elicit a wide range of emotions within an immersive VR environment. VR was chosen for its ability to provide controlled conditions and overcome the limitations of current mobile sensor technologies. Physiological data streams from various sensors were integrated for comprehensive emotional analysis. ElectroEncephaloGraphy (EEG) data revealed brain activity linked to emotional states, while eye tracking data provided insights into gaze direction, pupil dilation, and eye movement—factors correlated with cognitive and emotional processes. Peripheral signals, including heart rate variability, ElectroDermal Activity (EDA), and body temperature, were captured via wearable sensors to enrich the dataset. Machine learning models, such as XGBoost, CatBoost, Multilayer Perceptron, Gradient Boosting, and LightGBM, were employed to predict participants’ emotional states. Evaluation metrics, including accuracy, precision, recall, and F1 scores, demonstrated the robustness and precision of the proposed VR-based multi-sensor fusion approach. This research presents a novel approach to emotion recognition, bridging gaps in traditional methods by integrating VR, multi-sensor fusion, and machine learning.