Expanding Intra-Modal and Compressing Inter-Modal Embedding Space for Incomplete Multimodal Emotion Recognition

Abstract

Multimodal Emotion Recognition (MER) often suffers from incomplete modality inputs caused by sensor failures or privacy constraints. Recent studies generally fall into two categories: reconstruction-based methods and joint representation learning. The former recovers missing modalities from available ones, while the latter projects modalities into a shared latent space. However, these methods remain sensitive to single-modality variations and struggle to achieve effective alignment due to the gap between homogeneous intra-modal and heterogeneous inter-modal embedding spaces. To address these limitations, we propose a novel ECI framework that simultaneously Expands intra-modal and Compresses Inter-modal embedding spaces. ECI consists of three key components: Inter-modal Volume Alignment (IVA), HyperGraph-augmented Self-attention (HGS), and Intra-modal Distribution Normalization (IDN). Firstly, IVA reduces geometric inconsistency by minimizing the kernelized volume they form, encouraging compact and well-aligned cross-modal distributions. Secondly, HGS integrates hypergraph convolution with self-attention to capture high-order dependencies and pairwise interactions, enabling effective reconstruction of missing modalities. Finally, IDN applies contrastive suppression to expand feature distributions within intra-modal spaces to avoid overfitting and strengthen the discriminative power of each modality. Consequently, ECI achieves global alignment and complementary information flow while enhancing individual modality discriminability. Experiments on multiple benchmark datasets demonstrate that ECI achieves state-of-the-art performance across various missing modality scenarios.