MMPF: Multimodal Purification Fusion for Automatic Depression Detection

Abstract

Depression is a common mental disorder that requires objective and valid assessment tools. However, purely data-driven methods cannot satisfy the clinical diagnostic criteria for automatic depression detection (ADD), and the instability and heterogeneity of multimodal data have not been fully resolved. Therefore, we propose a novel auxiliary tool for ADD based on multimodal purification fusion (MMPF). Initially, a prior constraint gating (PCG) strategy is used to inject doctors’ constraints into depression data to guide and constrain the learning process. Then, we introduce text and audio encoders to extract unpurified features from preprocessed depression data. Afterward, multimodal purification refinement is proposed to extract unintersected common and specific features from unpurified features, generating purified features. Meanwhile, we leverage a multiperspective contrastive learning (MCL) strategy to enhance unpurified and purified features. Finally, modality interaction (MI) based on the transformer is proposed to conduct multimodal fusion. A dynamic corrective learning (DCL) strategy is introduced to tackle modality imbalances and inconsistent sentiment. MMPF is evaluated on the Distress Analysis Interview Corpus Wizard of Oz and performs promisingly in unimodal and multimodal depression detection, indicating its significant role in ADD.