ER-MRL: Emotion Recognition based on Multimodal Representation Learning

Abstract

In recent years, emotion recognition technology has been widely used in emotion change perception and mental illness diagnosis. Previous methods are mainly based on single-task learning strategies, which are unable to fuse multimodal features and remove redundant information. This paper proposes an emotion recognition model ER-MRL, which is based on multimodal representation learning. ER-MRL vectorizes the multimodal emotion data through encoders based on neural networks. The gate mechanism is used for multimodal feature selection. On this basis, ER-MRL calculates the modality specific and modality invariant representation for each emotion category. The Transformer model and multihead self-attention layer are applied to multimodal feature fusion. ER-MRL figures out the prediction result through the tower layer based on fully connected neural networks. Experimental results on the CMU-MOSI dataset show that ER-MRL has better performance on emotion recognition than previous methods.