Enhancing emotion recognition through multi-modal data fusion and graph neural networks

Abstract

In this paper, a novel emotion detection system is proposed based on Graph Neural Network (GNN) architecture, which is used to integrate and learn from multiple data sets (EEG, face expression, physiological signals). The proposed GNN is able to learn about interactions between multiple modalities, so as to extract a single picture of emotion categorization. This model is very good and gets 91.25 % accuracy, 91.26 % precision, 91.25 % recall and 91.25 % F1-score. Moreover, the proposed GNN is a sensible trade-off between speed and precision, with a calculation time of 163 ms. The Proposed GNN is better, primarily due to its ability to represent complex relations between multi-modal inputs, thereby improving its real-time emotional state recognition and classification performance. The proposed GNN demonstrates its suitability for powerful emotion detection by outperforming all models in classification precision and multi-modal data fusion, surpassing traditional models such as SVM, KNN, CCA, CNN, and RNN. The Proposed GNN consistently proves to be the most accurate and robust solution, having been the most dominant technique in emotion detection, despite CNN and RNN achieving slightly better results.