Enhanced Human-Machine Interactive Learning for Multimodal Emotion Recognition in Dialogue System

Emotion recognition has been well researched in mono-modality in the past decade. However, people express their emotion or feelings naturally via more than one modalities like voice, facial expressions, text, and behaviors. In this paper, we propose a new method to model deep interactive learning and dual modalities (e.g., speech and text) to conduct multimodal emotion recognition. An unsupervised triplet-loss objective function is constructed to learn representation of emotional information from speech audio. We extract text emotional feature representation by transfer learning of text-to-text embedding from T5 pre-trained model. Human-machine interaction like user feedback plays a vital role in improve multimodal emotion recognition in dialogue system. Deep interactive learning model is constructed by explicit and implicit feedback. Human-machine interactive learning enhanced transformer model can achieve higher levels of accuracy and precision than their non-interactive counterparts.