3WD-DRT: A three-way decision enhanced dynamic routing transformer for cost-sensitive multimodal sentiment analysis

Abstract

Accurately interpreting human emotion from language, facial expressions, and vocal tones remains a fundamental challenge in artificial intelligence. Current Multimodal Sentiment Analysis (MSA) models often struggle with two key issues. First, their static fusion strategies fail to handle conflicting modalities, such as sarcasm. Second, their standard loss functions ignore the asymmetric risks of severe misjudgments. To address these limitations, we propose the Three-Way Decision Enhanced Dynamic Routing Transformer (3WD-DRT), a framework operating on a "quality-aware, decision-driven" principle. It dynamically assesses each modality’s quality using a three-way decision gate, implemented via a dedicated MLP, to partition information into acceptance, deferment, or rejection pathways. This enables the model to amplify informative signals, moderately scale uncertain ones (deferment), and attenuate noisy or misleading ones. We also introduce a novel cost-sensitive loss function that imposes greater penalties on major semantic errors, such as polarity misclassifications. This approach better aligns the model’s training objective with human perception. Extensive experiments on CH-SIMS, CH-SIMSv2, MOSI, and MOSEI datasets show that 3WD-DRT consistently outperforms state-of-the-art methods, setting new benchmarks with F1-scores of 87.08 % on MOSI and 88.26 % on MOSEI. This work provides a robust solution for MSA, fostering more nuanced and reliable emotionally-aware AI systems.