Factorization-Based Information Reconstruction for Enhancing Missing Modality Robustness

Abstract

In recent years, Multimodal Sentiment Analysis (MSA) has emerged as a prominent research area, utilizing multiple signals to better understand human sentiment. Previous studies in MSA have primarily concentrated on performing interaction and fusion with complete signals. However, they have overlooked the issue of missing signals, which commonly arise in real-world scenarios due to factors such as occlusion, privacy concerns, and device malfunctions, leading to reduced generalizability. To this end, we propose a Factorization-based Information Reconstruction Framework (FIRF) to mitigate the modality missing problem in the MSA task. Specifically, we propose a fine-grained complementary factorization module that factorizes modality into synergistic, modality-heterogeneous, and noisy representations and design elaborate constraint paradigms for representation learning. Furthermore, we design a distribution calibration self-distillation module that fully recovers the missing semantics by utilizing bidirectional knowledge transfer. Comprehensive experiments on two datasets indicate that FIRF has a significant performance advantage over previous methods with uncertain missing modalities.