TF-BERT: Tensor-based fusion BERT for multimodal sentiment analysis

Abstract

Multimodal Sentiment Analysis (MSA) has gained significant attention due to the limitations of unimodal sentiment recognition in complex real-world applications. Traditional approaches typically focus on using the Transformer for fusion. However, these traditional approaches often fall short because the Transformer can only process two modalities simultaneously, leading to insufficient information exchange and potential loss of emotional data. To address the limitation of traditional Crossmodal Transformer models, which can only process two modalities at a time, we propose a novel Tensor-based Fusion BERT model (TF-BERT). The core of TF-BERT is the Tensor-based Crossmodal Fusion (TCF) module, which seamlessly integrates into the pre-trained BERT language model. By embedding the TCF module into multiple layers of BERT’s Transformer, we progressively achieve dynamic complementation between different modalities. Additionally, we designed the Tensor-based Crossmodal Transformer (TCT) module, which introduces a tensor-based Transformer mechanism capable of simultaneously processing three different modalities. This allows for comprehensive information exchange between the target modality and the other two source modalities, thus strengthening the representation of the target modality. The TCT overcomes the limitation of existing the Crossmodal Transformer structures, which can only handle relationships between two modalities. Furthermore, to validate the effectiveness of TF-BERT, we conducted extensive experiments on the CMU-MOSI and CMU-MOSEI datasets. TF-BERT not only achieved the best results across most metrics compared to recent state-of-the-art baselines, but also demonstrated the effectiveness of its two modules through ablation studies. The findings suggest that TF-BERT effectively addresses the limitations of previous models by progressively integrating and simultaneously capturing complex emotional interactions across all modalities.