CMFF: A cross-modal multi-layer feature fusion network for multimodal sentiment analysis

Multimodal sentiment analysis seeks to interpret speaker sentiment by integrating information from multiple modalities, typically text and audio. While existing methods often focus on fusing deep-layer features extracted from the final stages of unimodal encoders, they may overlook crucial fine-grained information present in shallow-layer features (e.g., subtle phonetic variations or basic syntactic structures) relevant for nuanced sentiment understanding. Furthermore, effectively fusing features from different modalities presents the dual challenges of dynamically weighting each modality’s contribution and accommodating their inherent data heterogeneity. To address these limitations, we propose a novel Cross-modal Multi-layer Feature Fusion (CMFF) network. CMFF explicitly leverages the hierarchical information contained in both shallow-layer and deep-layer features from text and audio modalities. It employs multi-head cross-modal attention mechanisms within its fusion layers to facilitate interaction across feature layers and modalities. Crucially, CMFF incorporates a Mixture of Gated Experts (MoGE) network within these fusion layers. The MoGE utilizes modality-specific expert sub-networks, each tailored to process the distinct characteristics of text or audio data, thereby directly addressing data heterogeneity. Concurrently, each expert employs an internal gated feed-forward mechanism. This allows the model to dynamically control the information flow for each feature vector, effectively learning to weigh the importance of different feature dimensions from each layer and modality based on the input context. Extensive experiments conducted on the benchmark CMU-MOSI and CMU-MOSEI datasets demonstrate that the proposed CMFF model achieves competitive or superior performance compared to state-of-the-art methods across various standard evaluation metrics.