Visually multimodal depression assessment based on key questions with weighted multi-task learning

In recent years, depression has received attention due to its high prevalence and high risk of suicide. In contrast, the increased pressure on health care and the shortage of mental health professionals have led to the failure to detect and intervene in depression promptly. To solve the above problems, we propose a visual multi-modal fusion network for depression assessment based on weighted multi-task learning (WMTL). First, the visual cues of different modalities are collected from the subjects when they answer key questions in the simulated interview to mitigate redundancy. Afterward, spatial attention-based feature embedding modules are proposed to extract depression-aware features from different visual cues. Finally, a hierarchical weighted attention fusion (HAF) module is presented to fuse the depression-aware features from different modalities and facilitate depression assessment. Comprehensive evaluations are conducted on the benchmarking DAIC-WOZ. Experimental results show that the proposed method performs well in assessing depression, with an average accuracy of 76.96% for ten questions and an F1 score of 0.85. The high performance also indicates a strong correlation between key questions in the interview and depression levels.