CIEG-Net: Context Information Enhanced Gated Network for multimodal sentiment analysis

Abstract

Multimodal sentiment analysis is a widely studied field aimed at recognizing sentiment information through multiple modalities. The primary challenge in this field lies in developing high-quality fusion frameworks that effectively address the heterogeneity among different modalities and the issue of feature loss during the fusion process. However, existing research has primarily focused on cross-modal fusion, with relatively little attention paid to the sentiment semantics conveyed by context information. In this paper, we propose the Context Information Enhanced Gated Network (CIEG-Net), a novel fusion network that enhances multimodal fusion by incorporating context information from the input modalities. Specifically, we first construct a context information enhanced module to obtain the input and corresponding context information for the text and audio modalities. Then, we designed a fusion network module that facilitates the fusion between the text–audio modality and their respective text-context and audio-context information. Finally, we propose a gated network module that dynamically adjusts the weights of each modality and its context information, further strengthening multimodal fusion and attempting to recover missing features. We evaluate the proposed model on three publicly available multimodal sentiment analysis datasets: CMU-MOSI, CMU-MOSEI, and CH-SIMS. Experimental results show that our model significantly outperforms the current SOTA models.