An early fusion approach for multimodal emotion recognition using deep recurrent networks

In this paper we compare different strategies for handling incomplete data and different classification architectures for emotion recognition from multimodal data, using an early fusion approach. In order to allow the different modalities to complement each other at feature level, the initial task was to align the data at the same frame rate. The source data possessed a high degree of incompleteness, which we addressed by different imputation approaches. Since the data was missing in blocks, we found that the best performing approach was to replace missing values with zeros. For the classification model, we experimented with LSTM and GRU networks, in both unidirectional and bidirectional flavors, and various hyper-parameter settings. We found that a bidirectional GRU model trained using a smaller batch size and more aggressive dropout produced the best classification performance.