MMTrans-MT: A Framework for Multimodal Emotion Recognition Using Multitask Learning

Abstract

With the development of deep learning, emotion recognition tasks are more inclined to use multimodal data and adequate supervised information to improve accuracy. In this work, MMTrans-MT (Multimodal Transformer-Multitask), the framework for multimodal emotion recognition using multitask learning is proposed. It has three modules: modalities representation module, multimodal fusion module, and multitask output module. Three modalities, i.e, words, audio and video, are comprehensively utilized to carry out emotion recognition by a simple but efficient fusion model based on Transformer. As for multitask learning, the two tasks are defined as categorical emotion classification and dimensional emotion regression. Considering a potential mapping relationship between two kinds of emotion model, multitask learning is adopted to make the two tasks promote each other and improve recognition accuracy. We conduct experiments on CMU-MOSEI and IEMOCAP datasets. Comprehensive experiments show that the accuracy of recognition using multimodal information is higher than that using unimodal information. Adopting multitask learning promotes the performance of emotion recognition.