Dataset-aware Utopia modality contribution for imbalanced multimodal learning

Multimodal imbalance is a critical issue in multimodal learning research. Over recent years, numerous modulation strategies have been proposed, with a core focus on minimizing disparities in contributions across modalities. However, we observe that in many datasets, the contribution proportions of modalities are inherently unequal. Consequently, diagnosing multimodal imbalanced learning based on the criterion of equal contributions and optimizing models by minimizing modality contribution disparities often result in suboptimal performance. To address this issue, we propose the concept of “Utopia Contribution”, which estimates the utopia contribution distribution of each modality based on dataset-specific characteristics. This distribution serves as the optimization objective for modulation strategies, facilitating the comprehensive exploitation of information from all modalities. Specifically, based on the principle of population risk, we estimate the utopia contribution distribution of modalities in the given dataset by analyzing the impact of each modality’s presence or absence on model predictions. Additionally, to enhance the generalizability of our method, we further propose a model-agnostic approach based on mutual information to estimate the factual contribution distribution of each modality. During training, we employ KL divergence (Kullback–Leibler divergence) to align the factual contribution distribution with the utopia contribution distribution. Extensive experiments on three benchmark datasets - IEMOCAP, CMU-MOSEI, and AVE - demonstrate the rationality, reliability, and effectiveness of our method.