CR-GAC: Cross-modal Recombination via Graph-Attention Collaborative Optimization for multimodal sentiment analysis

Abstract

Multimodal sentiment analysis integrates linguistic, audio, and visual modalities for predicting human emotional states. However, current algorithms encounter three challenges: limitations in adjacency matrix modeling, noise interference and modality imbalances in cross-modal attention, and inefficient cross-modal feature alignment. To address these, we propose the Cross-modal Recombination via Graph-Attention Collaborative Optimization (CR-GAC) by unifying graph and sequence learning in a collaborative framework. Specifically, we first design the modality-adaptive Multimodal Graph Construction (MGC) to tackle the first challenge. For the linguistic modality, a local sparse graph based on a K-Nearest Neighbors-Radial Basis Function kernel is designed to preserve fine-grained semantics; for the audio and visual modalities, a low-rank representation method combined with nuclear norm regularization is designed to capture latent cross-sample structures via singular value decomposition, while suppressing noise interference. Modalities that have been processed are then input into graph attention networks to achieve higher-order feature aggregation. Next, we construct the Language-guided Hierarchical Cross-modal Interaction (LHCI) to tackle the second challenge, which leverages bidirectional cross-modal attention and multi-level Transformer blocks to hierarchically enhance feature representations. Subsequently, the High-level Multimodal Feature Container (HMFC) iteratively accumulates multi-grained semantics, providing a high-level feature pool for fusion. Finally, the dynamic matching-based High-level Feature Recombination (HFR) is designed to tackle the third challenge, which uses the linguistic feature as an anchor to achieve semantically controllable explicit alignment and flexible implicit alignment by matching the most relevant features. Experimental results show our model achieves state-of-the-art performance on CMU-MOSI and CMU-MOSEI datasets, and demonstrates generalization capability on CH-SIMS dataset.