AMGFM: A multimodal fusion model for depression identification based on adversarial metric learning.

Abstract

Early detection of depression is essential for timely intervention, yet objective assessment remains difficult. Recent studies have explored multimodal physiological signals, including electroencephalography (EEG), eye movement, and pupil dynamics, to improve depression recognition. However, existing multimodal approaches still face two major limitations: insufficient alignment across heterogeneous modalities, which results in modality-dependent feature representations, and limited systematic analysis of how emotional stimuli and cross-modal interactions affect recognition performance, thereby constraining both accuracy and interpretability. To address these issues, we propose AMGFM, an Adversarial-Metric Graph Fusion Model for multimodal depression recognition. AMGFM uses adversarial learning to reduce global distribution discrepancies across modalities and metric learning to enforce intra-class compactness and inter-class separability in a shared latent space. In addition, a hierarchical graph fusion module explicitly models and adaptively integrates unimodal, bimodal, and trimodal interactions to exploit complementary information. Experiments on EFVP and AADP demonstrate that AMGFM consistently outperforms unimodal baselines and representative multimodal fusion approaches, achieving up to 90.33% accuracy. Further analyses indicate that neutral stimuli provide stronger discriminative cues than emotional stimuli, and that EEG serves as an effective target modality for aligning other signals.