Multimodal Fusion of EEG and Audio Spectrogram for Major Depressive Disorder Recognition Using Modified DenseNet121.

IF 2.7 3区医学 Q3 NEUROSCIENCES

Brain Sciences Pub Date : 2024-10-15 DOI:10.3390/brainsci14101018

Musyyab Yousufi, Robertas Damaševičius, Rytis Maskeliūnas

{"title":"Multimodal Fusion of EEG and Audio Spectrogram for Major Depressive Disorder Recognition Using Modified DenseNet121.","authors":"Musyyab Yousufi, Robertas Damaševičius, Rytis Maskeliūnas","doi":"10.3390/brainsci14101018","DOIUrl":null,"url":null,"abstract":"Background/objectives: This study investigates the classification of Major Depressive Disorder (MDD) using electroencephalography (EEG) Short-Time Fourier-Transform (STFT) spectrograms and audio Mel-spectrogram data of 52 subjects. The objective is to develop a multimodal classification model that integrates audio and EEG data to accurately identify depressive tendencies.Methods: We utilized the Multimodal open dataset for Mental Disorder Analysis (MODMA) and trained a pre-trained Densenet121 model using transfer learning. Features from both the EEG and audio modalities were extracted and concatenated before being passed through the final classification layer. Additionally, an ablation study was conducted on both datasets separately.Results: The proposed multimodal classification model demonstrated superior performance compared to existing methods, achieving an Accuracy of 97.53%, Precision of 98.20%, F1 Score of 97.76%, and Recall of 97.32%. A confusion matrix was also used to evaluate the model's effectiveness.Conclusions: The paper presents a robust multimodal classification approach that outperforms state-of-the-art methods with potential application in clinical diagnostics for depression assessment.","PeriodicalId":9095,"journal":{"name":"Brain Sciences","volume":"14 10","pages":""},"PeriodicalIF":2.7000,"publicationDate":"2024-10-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11505707/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Brain Sciences","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.3390/brainsci14101018","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"NEUROSCIENCES","Score":null,"Total":0}

引用次数: 0

Abstract

Background/objectives: This study investigates the classification of Major Depressive Disorder (MDD) using electroencephalography (EEG) Short-Time Fourier-Transform (STFT) spectrograms and audio Mel-spectrogram data of 52 subjects. The objective is to develop a multimodal classification model that integrates audio and EEG data to accurately identify depressive tendencies.

Methods: We utilized the Multimodal open dataset for Mental Disorder Analysis (MODMA) and trained a pre-trained Densenet121 model using transfer learning. Features from both the EEG and audio modalities were extracted and concatenated before being passed through the final classification layer. Additionally, an ablation study was conducted on both datasets separately.

Results: The proposed multimodal classification model demonstrated superior performance compared to existing methods, achieving an Accuracy of 97.53%, Precision of 98.20%, F1 Score of 97.76%, and Recall of 97.32%. A confusion matrix was also used to evaluate the model's effectiveness.

Conclusions: The paper presents a robust multimodal classification approach that outperforms state-of-the-art methods with potential application in clinical diagnostics for depression assessment.

查看原文本刊更多论文

利用修改后的 DenseNet 121 多模态融合脑电图和音频频谱图识别重度抑郁障碍。

背景/目的：本研究利用 52 名受试者的脑电图（EEG）短时傅立叶变换（STFT）频谱图和音频 Mel 频谱图数据，对重度抑郁症（MDD）进行分类。目的是开发一种多模态分类模型，整合音频和脑电图数据，准确识别抑郁倾向：我们利用精神障碍分析多模态开放数据集（MODMA），并通过迁移学习训练了一个预先训练好的 Densenet121 模型。在通过最终分类层之前，我们提取了脑电图和音频模式的特征并进行了串联。此外，还对两个数据集分别进行了消融研究：与现有方法相比，所提出的多模态分类模型表现优异，准确率达 97.53%，精确率达 98.20%，F1 分数达 97.76%，召回率达 97.32%。混淆矩阵也被用来评估模型的有效性：本文提出了一种稳健的多模态分类方法，其性能优于最先进的方法，有望应用于抑郁症评估的临床诊断。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Brain Sciences Neuroscience-General Neuroscience

CiteScore

4.80

自引率

9.10%

发文量

1472

审稿时长

18.71 days

期刊介绍： Brain Sciences (ISSN 2076-3425) is a peer-reviewed scientific journal that publishes original articles, critical reviews, research notes and short communications in the areas of cognitive neuroscience, developmental neuroscience, molecular and cellular neuroscience, neural engineering, neuroimaging, neurolinguistics, neuropathy, systems neuroscience, and theoretical and computational neuroscience. Our aim is to encourage scientists to publish their experimental and theoretical results in as much detail as possible. There is no restriction on the length of the papers. The full experimental details must be provided so that the results can be reproduced. Electronic files or software regarding the full details of the calculation and experimental procedure, if unable to be published in a normal way, can be deposited as supplementary material.