Depression level prediction via textual and acoustic analysis

Abstract

Extensive research on automatic depression diagnosis has utilized video data to capture related cues, but data collection is challenging because of privacy concerns. By contrast, voice data offer a less-intrusive assessment method and can be analyzed for features such as simple tones, the expression of negative emotions, and a focus on oneself. Recent advancements in multimodal depression-level prediction using speech and text data have gained traction, but most studies overlook the temporal alignment of these modalities, limiting their analysis of the interaction between speech content and intonation. To overcome these limitations, this study introduces timestamp-integrated multimodal encoding for depression (TIMEX-D) which synchronizes the acoustic features of human speech with corresponding text data to predict depression levels on the basis of their relationship. TIMEX-D comprises three main components: a timestamp extraction block that extracts timestamps from speech and text, a multimodal encoding block that extends positional encoding from transformers to mimic human speech recognition, and a depression analysis block that predicts depression levels while reducing model complexity compared with existing transformers. In experiments using the DAIC-WOZ and EDAIC datasets, TIMEX-D achieved accuracies of 99.17 % and 99.81 %, respectively, outperforming previous methods by approximately 13 %. The effectiveness of TIMEX-D in predicting depression levels can enhance mental health diagnostics and monitoring across various contexts.