$$\text {H}^2\text {CAN}$$ : heterogeneous hypergraph attention network with counterfactual learning for multimodal sentiment analysis

Abstract

Multimodal sentiment analysis (MSA) has garnered significant attention for its immense potential in human-computer interaction. While cross-modality attention mechanisms are widely used in MSA to capture inter-modality interactions, existing methods are limited to pairwise interactions between two modalities. Additionally, these methods can not utilize the causal relationship to guide attention learning, making them susceptible to bias information. To address these limitations, we introduce a novel method called Heterogeneous Hypergraph Attention Network with Counterfactual Learning \((\text {H}^2\text {CAN}).\) The method constructs a heterogeneous hypergraph based on sentiment expression characteristics and employs Heterogeneous Hypergraph Attention Networks (HHGAT) to capture interactions beyond pairwise constraints. Furthermore, it mitigates the effects of bias through a Counterfactual Intervention Task (CIT). Our model comprises two main branches: hypergraph fusion and counterfactual fusion. The former uses HHGAT to capture inter-modality interactions, while the latter constructs a counterfactual world using Gaussian distribution and additional weighting for the biased modality. The CIT leverages causal inference to maximize the prediction discrepancy between the two branches, guiding attention learning in the hypergraph fusion branch. We utilize unimodal labels to help the model adaptively identify the biased modality, thereby enhancing the handling of bias information. Experiments on three mainstream datasets demonstrate that \(\text {H}^2\text {CAN}\) sets a new benchmark.