Extending Multimodal Emotion Recognition with Biological Signals: Presenting a Novel Dataset and Recent Findings

Abstract

Multimodal fusion has shown great promise in recent literature, particularly for audio dominant tasks. In this talk, we outline a the finding from a recently developed multimodal dataset, and discuss the promise of fusing biological signals with speech for continuous recognition of the emotional dimensions of valence and arousal in the context of public speaking. As well as this, we discuss the advantage of cross-language (German and English) analysis by training language-independent models and testing them on speech from various native and non-native groupings. For the emotion recognition task used as a case study, a Long Short-Term Memory - Recurrent Neural Network (LSTM-RNN) architecture with a self-attention layer is used.