Semi-Tensor Product based Multi-modal Fusion Method for Emotion Recognition

Abstract

Emotion recognition has been an important research topic in the field of human-computer interaction. Multi-modal emotion recognition makes full use of the complementarity of different modalities, which has greater advantages than single-modal emotion recognition. Traditional methods have low prediction performance and limited dimension of tensor fusion caused by inadequate multi-modal information fusion. In this paper, we proposed the semi-tensor product and attention based low-rank multi-modal fusion network (STALMF) for emotion recognition. We first use the semi-tensor product to effectively combine acoustic and language features. The self-attention module is then used for modeling the temporal dependencies of the fused features. Finally, the low-rank multi-modal fusion module is adopted to adequately fuse the information between the fused feature and the individual feature. We conducted our proposed method on the IEMOCAP dataset. The proposed method achieves an averaged F1 score of 82.4% and accuracy of 83.0%, outperforming the comparative methods. Experimental results show that the proposed method can effectively fuse multi-modal information by introducing the semi-tensor product, self-attention mechanism and low-rank multi-modal fusion module.