A sequential mixing fusion network for enhanced feature representations in multimodal sentiment analysis

Abstract

Multimodal sentiment analysis exploits multiple modalities to understand a user’s sentiment state from video content. Recent work in this area integrates features derived from different modalities. However, current multimodal sentiment datasets are typically small with limited cross-modal interaction diversity, for which simple feature fusion mechanisms can lead to modality dependence and model overfitting. Consequently, how to augment diverse cross-modal samples and use non-verbal modalities to dynamically enhance text feature representations is still under-explored. In this paper, we propose a sequential mixing fusion network to tackle this research challenge. Using speech text content as a primary source, we design a sequential fusion strategy to maximise the feature expressiveness enhanced by auxiliary modalities, namely facial movements and audio features, and a random feature-level mixing algorithm to augment diverse cross-modality interactions. Experimental results on three benchmark datasets show that our proposed approach significantly outperforms current state-of-the-art methods, while demonstrating strong robustness capability when dealing with a missing modality.