ST-SHAP:用于情绪脑电图表征学习和解码的分层可解释注意力网络

IF 2.7 4区 医学 Q2 BIOCHEMICAL RESEARCH METHODS
Minmin Miao , Jin Liang , Zhenzhen Sheng , Wenzhe Liu , Baoguo Xu , Wenjun Hu
{"title":"ST-SHAP:用于情绪脑电图表征学习和解码的分层可解释注意力网络","authors":"Minmin Miao ,&nbsp;Jin Liang ,&nbsp;Zhenzhen Sheng ,&nbsp;Wenzhe Liu ,&nbsp;Baoguo Xu ,&nbsp;Wenjun Hu","doi":"10.1016/j.jneumeth.2024.110317","DOIUrl":null,"url":null,"abstract":"<div><h3>Background:</h3><div>Emotion recognition using electroencephalogram (EEG) has become a research hotspot in the field of human–computer interaction, how to sufficiently learn complex spatial–temporal representations of emotional EEG data and obtain explainable model prediction results are still great challenges.</div></div><div><h3>New method</h3><div>In this study, a novel hierarchical and explainable attention network ST-SHAP which combines the Swin Transformer (ST) and SHapley Additive exPlanations (SHAP) technique is proposed for automatic emotional EEG classification. Firstly, a 3D spatial–temporal feature of emotional EEG data is generated via frequency band filtering, temporal segmentation, spatial mapping, and interpolation to fully preserve important spatial–temporal-frequency characteristics. Secondly, a hierarchical attention network is devised to sufficiently learn an abstract spatial–temporal representation of emotional EEG and perform classification. Concretely, in this decoding model, the W-MSA module is used for modeling correlations within local windows, the SW-MSA module allows for information interactions between different local windows, and the patch merging module further facilitates local-to-global multiscale modeling. Finally, the SHAP method is utilized to discover important brain regions for emotion processing and improve the explainability of the Swin Transformer model.</div></div><div><h3>Results:</h3><div>Two benchmark datasets, namely SEED and DREAMER, are used for classification performance evaluation. In the subject-dependent experiments, for SEED dataset, ST-SHAP achieves an average accuracy of 97.18%, while for DREAMER dataset, the average accuracy is 96.06% and 95.98% on arousal and valence dimension respectively. In addition, important brain regions that conform to prior knowledge of neurophysiology are discovered via a data-driven approach for both datasets.</div></div><div><h3>Comparison with existing methods:</h3><div>In terms of subject-dependent and subject-independent emotional EEG decoding accuracies, our method outperforms several closely related existing methods.</div></div><div><h3>Conclusion:</h3><div>These experimental results fully prove the effectiveness and superiority of our proposed algorithm.</div></div>","PeriodicalId":16415,"journal":{"name":"Journal of Neuroscience Methods","volume":"414 ","pages":"Article 110317"},"PeriodicalIF":2.7000,"publicationDate":"2024-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"ST-SHAP: A hierarchical and explainable attention network for emotional EEG representation learning and decoding\",\"authors\":\"Minmin Miao ,&nbsp;Jin Liang ,&nbsp;Zhenzhen Sheng ,&nbsp;Wenzhe Liu ,&nbsp;Baoguo Xu ,&nbsp;Wenjun Hu\",\"doi\":\"10.1016/j.jneumeth.2024.110317\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><h3>Background:</h3><div>Emotion recognition using electroencephalogram (EEG) has become a research hotspot in the field of human–computer interaction, how to sufficiently learn complex spatial–temporal representations of emotional EEG data and obtain explainable model prediction results are still great challenges.</div></div><div><h3>New method</h3><div>In this study, a novel hierarchical and explainable attention network ST-SHAP which combines the Swin Transformer (ST) and SHapley Additive exPlanations (SHAP) technique is proposed for automatic emotional EEG classification. Firstly, a 3D spatial–temporal feature of emotional EEG data is generated via frequency band filtering, temporal segmentation, spatial mapping, and interpolation to fully preserve important spatial–temporal-frequency characteristics. Secondly, a hierarchical attention network is devised to sufficiently learn an abstract spatial–temporal representation of emotional EEG and perform classification. Concretely, in this decoding model, the W-MSA module is used for modeling correlations within local windows, the SW-MSA module allows for information interactions between different local windows, and the patch merging module further facilitates local-to-global multiscale modeling. Finally, the SHAP method is utilized to discover important brain regions for emotion processing and improve the explainability of the Swin Transformer model.</div></div><div><h3>Results:</h3><div>Two benchmark datasets, namely SEED and DREAMER, are used for classification performance evaluation. In the subject-dependent experiments, for SEED dataset, ST-SHAP achieves an average accuracy of 97.18%, while for DREAMER dataset, the average accuracy is 96.06% and 95.98% on arousal and valence dimension respectively. In addition, important brain regions that conform to prior knowledge of neurophysiology are discovered via a data-driven approach for both datasets.</div></div><div><h3>Comparison with existing methods:</h3><div>In terms of subject-dependent and subject-independent emotional EEG decoding accuracies, our method outperforms several closely related existing methods.</div></div><div><h3>Conclusion:</h3><div>These experimental results fully prove the effectiveness and superiority of our proposed algorithm.</div></div>\",\"PeriodicalId\":16415,\"journal\":{\"name\":\"Journal of Neuroscience Methods\",\"volume\":\"414 \",\"pages\":\"Article 110317\"},\"PeriodicalIF\":2.7000,\"publicationDate\":\"2024-11-12\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of Neuroscience Methods\",\"FirstCategoryId\":\"3\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S0165027024002620\",\"RegionNum\":4,\"RegionCategory\":\"医学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q2\",\"JCRName\":\"BIOCHEMICAL RESEARCH METHODS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Neuroscience Methods","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0165027024002620","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"BIOCHEMICAL RESEARCH METHODS","Score":null,"Total":0}
引用次数: 0

摘要

研究背景利用脑电图(EEG)进行情绪识别已成为人机交互领域的研究热点,如何充分学习情绪EEG数据的复杂时空表征并获得可解释的模型预测结果仍是巨大挑战:本研究提出了一种新颖的分层可解释注意力网络 ST-SHAP,它结合了 Swin Transformer(ST)和 SHapley Additive exPlanations(SHAP)技术,用于自动情绪脑电图分类。首先,通过频带滤波、时间分割、空间映射和插值生成情绪脑电数据的三维时空特征,以充分保留重要的时空频率特性。其次,设计分层注意力网络,以充分学习情绪脑电图的抽象时空表征并进行分类。具体来说,在该解码模型中,W-MSA 模块用于对局部窗口内的相关性建模,SW-MSA 模块允许不同局部窗口之间的信息交互,而补丁合并模块则进一步促进了局部到全局的多尺度建模。最后,SHAP方法用于发现情绪处理的重要脑区,并提高Swin Transformer模型的可解释性:结果:两个基准数据集(即 SEED 和 DREAMER)用于分类性能评估。在受试者依赖性实验中,ST-SHAP 在 SEED 数据集上的平均准确率为 97.18%,而在 DREAMER 数据集上,ST-SHAP 在唤醒维度和价值维度上的平均准确率分别为 96.06% 和 95.98%。此外,在这两个数据集上,还通过数据驱动方法发现了符合神经生理学先验知识的导入脑区:与现有方法的比较:在与主体相关和与主体无关的情绪脑电图解码准确度方面,我们的方法优于几种密切相关的现有方法:这些实验结果充分证明了我们提出的算法的有效性和优越性。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
ST-SHAP: A hierarchical and explainable attention network for emotional EEG representation learning and decoding

Background:

Emotion recognition using electroencephalogram (EEG) has become a research hotspot in the field of human–computer interaction, how to sufficiently learn complex spatial–temporal representations of emotional EEG data and obtain explainable model prediction results are still great challenges.

New method

In this study, a novel hierarchical and explainable attention network ST-SHAP which combines the Swin Transformer (ST) and SHapley Additive exPlanations (SHAP) technique is proposed for automatic emotional EEG classification. Firstly, a 3D spatial–temporal feature of emotional EEG data is generated via frequency band filtering, temporal segmentation, spatial mapping, and interpolation to fully preserve important spatial–temporal-frequency characteristics. Secondly, a hierarchical attention network is devised to sufficiently learn an abstract spatial–temporal representation of emotional EEG and perform classification. Concretely, in this decoding model, the W-MSA module is used for modeling correlations within local windows, the SW-MSA module allows for information interactions between different local windows, and the patch merging module further facilitates local-to-global multiscale modeling. Finally, the SHAP method is utilized to discover important brain regions for emotion processing and improve the explainability of the Swin Transformer model.

Results:

Two benchmark datasets, namely SEED and DREAMER, are used for classification performance evaluation. In the subject-dependent experiments, for SEED dataset, ST-SHAP achieves an average accuracy of 97.18%, while for DREAMER dataset, the average accuracy is 96.06% and 95.98% on arousal and valence dimension respectively. In addition, important brain regions that conform to prior knowledge of neurophysiology are discovered via a data-driven approach for both datasets.

Comparison with existing methods:

In terms of subject-dependent and subject-independent emotional EEG decoding accuracies, our method outperforms several closely related existing methods.

Conclusion:

These experimental results fully prove the effectiveness and superiority of our proposed algorithm.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
Journal of Neuroscience Methods
Journal of Neuroscience Methods 医学-神经科学
CiteScore
7.10
自引率
3.30%
发文量
226
审稿时长
52 days
期刊介绍: The Journal of Neuroscience Methods publishes papers that describe new methods that are specifically for neuroscience research conducted in invertebrates, vertebrates or in man. Major methodological improvements or important refinements of established neuroscience methods are also considered for publication. The Journal''s Scope includes all aspects of contemporary neuroscience research, including anatomical, behavioural, biochemical, cellular, computational, molecular, invasive and non-invasive imaging, optogenetic, and physiological research investigations.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信