AC2Net: Hybrid attention convolution and compression fusion network for multimodal emotion recognition

The rapid development of human-computer interaction systems can help computers better understand human intentions. How to comprehensively and accurately recognize emotions is a key link in human-computer interaction. However, the current research on multimodal sentiment analysis tends to extract the information of a single modal separately and then simply combine the features of each mode. In this process, there is a lack of effective interaction between the features of each modal extraction, and the close relationship between the multi-modal data and the recognition task is ignored. In addition, the multimodal features are simply added and connected, ignoring the differences in the degree of correlation and contribution of different modal information to the final recognition result. In view of this, this paper proposes the Hybrid Attention Convolution and Compression Fusion Unit for Multimodal Emotion Recognition, namely AC2Net. We have designed a hybrid attention convolution model, which focuses on the interactive extraction of multi-modal features and can accurately capture key emotional perception features among three signal features, namely facial image sequence, EEG, and peripheral physiological signals. A compression fusion unit is designed to aggregate the multi-modal features, and the deep fusion of multi-modal heterogeneous information is realized by compressing and exciting the dense layers in the multi-modal branches. Finally, the proposed model was verified on the DEAP dataset, and the accuracy of valence and arousal dimension two-classification recognition reached 99.39%, 99.37%, and four-classification recognition accuracy reached 99.08%. Compared with the existing single-mode and multi-mode emotion recognition methods, the performance of this model is excellent.