A multi-modal emotion recognition method considering the contribution and redundancy of channels and the correlation and heterogeneity of modalities

IF 5.6 2区 工程技术 Q1 ENGINEERING, MULTIDISCIPLINARY
Yongxuan Wen, Wanzhong Chen
{"title":"A multi-modal emotion recognition method considering the contribution and redundancy of channels and the correlation and heterogeneity of modalities","authors":"Yongxuan Wen,&nbsp;Wanzhong Chen","doi":"10.1016/j.measurement.2025.119247","DOIUrl":null,"url":null,"abstract":"<div><div>Physiological signals could reflect individual true emotional state, and emotion recognition based on physiological signals is significant in the field of artificial intelligence. However, current multimodal emotion recognition methods used full channels, leading to data redundancy and hardware complexity, causing a waste of computing resources. In addition, existing feature fusion methods generally adopted a direct connection approach, lacking of mid-level alignment and interaction, which cannot effectively extract complementary features from multimodal information, thus affecting classification accuracy. To address the above-mentioned issues, this paper proposed a multimodal emotion recognition method based on both electroencephalogram signals (EEG) and peripheral physiological signals (PPS). First, we introduced a triple-weighted ReliefF-NMI channel selection (TWRNCS) to select channels for EEG signals where the triple weight of subject-feature-frequency band were considered, and the contribution and redundancy of EEG channels are screened in two stages. Secondly, we designed an adaptive feature extractor capable of automatically exacting features from multi-channel EEG and PPS. Additionally, we proposed a cross-modal hybrid attention module (CHAM) based on self-attention and cross-attention mechanisms, including intra-modality private pipelines and inter-modality common pipelines. The private pipelines used self-attention mechanisms to retain heterogeneous information of modalities, while the common pipelines used cross-attention and self-attention mechanisms to capture cross-modal correlations. Finally, the information from different modalities was fully integrated for classification. The experiments demonstrated that our model achieved accuracy of over 98% on the DEAP and MAHNOB-HCI datasets, which proved the superiority of this paper in emotion recognition tasks.</div></div>","PeriodicalId":18349,"journal":{"name":"Measurement","volume":"258 ","pages":"Article 119247"},"PeriodicalIF":5.6000,"publicationDate":"2025-10-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Measurement","FirstCategoryId":"5","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0263224125026065","RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ENGINEERING, MULTIDISCIPLINARY","Score":null,"Total":0}
引用次数: 0

Abstract

Physiological signals could reflect individual true emotional state, and emotion recognition based on physiological signals is significant in the field of artificial intelligence. However, current multimodal emotion recognition methods used full channels, leading to data redundancy and hardware complexity, causing a waste of computing resources. In addition, existing feature fusion methods generally adopted a direct connection approach, lacking of mid-level alignment and interaction, which cannot effectively extract complementary features from multimodal information, thus affecting classification accuracy. To address the above-mentioned issues, this paper proposed a multimodal emotion recognition method based on both electroencephalogram signals (EEG) and peripheral physiological signals (PPS). First, we introduced a triple-weighted ReliefF-NMI channel selection (TWRNCS) to select channels for EEG signals where the triple weight of subject-feature-frequency band were considered, and the contribution and redundancy of EEG channels are screened in two stages. Secondly, we designed an adaptive feature extractor capable of automatically exacting features from multi-channel EEG and PPS. Additionally, we proposed a cross-modal hybrid attention module (CHAM) based on self-attention and cross-attention mechanisms, including intra-modality private pipelines and inter-modality common pipelines. The private pipelines used self-attention mechanisms to retain heterogeneous information of modalities, while the common pipelines used cross-attention and self-attention mechanisms to capture cross-modal correlations. Finally, the information from different modalities was fully integrated for classification. The experiments demonstrated that our model achieved accuracy of over 98% on the DEAP and MAHNOB-HCI datasets, which proved the superiority of this paper in emotion recognition tasks.
一种考虑通道贡献和冗余、模态相关性和异质性的多模态情感识别方法
生理信号可以反映个体真实的情绪状态,基于生理信号的情绪识别在人工智能领域具有重要意义。然而,目前的多模态情感识别方法采用全通道,导致数据冗余和硬件复杂,造成了计算资源的浪费。此外,现有的特征融合方法一般采用直接连接的方式,缺乏中层对齐和交互,不能有效地从多模态信息中提取互补特征,影响分类精度。针对上述问题,本文提出了一种基于脑电图(EEG)和外周生理信号(PPS)的多模态情绪识别方法。首先,引入三加权relief - nmi信道选择(TWRNCS)方法,在考虑主体-特征-频带三权重的基础上对脑电信号进行信道选择,并分两个阶段对脑电信号信道的贡献和冗余进行筛选。其次,设计了一种自适应特征提取器,能够自动提取多通道EEG和PPS的特征;此外,我们提出了一种基于自注意和交叉注意机制的跨模态混合注意模块(CHAM),包括模态内私有管道和模态间公共管道。私有管道使用自关注机制来保留模态的异构信息,而公共管道使用交叉关注和自关注机制来捕获跨模态的相关性。最后,充分整合不同模式的信息进行分类。实验表明,我们的模型在DEAP和MAHNOB-HCI数据集上达到了98%以上的准确率,证明了本文在情绪识别任务中的优越性。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Measurement
Measurement 工程技术-工程:综合
CiteScore
10.20
自引率
12.50%
发文量
1589
审稿时长
12.1 months
期刊介绍: Contributions are invited on novel achievements in all fields of measurement and instrumentation science and technology. Authors are encouraged to submit novel material, whose ultimate goal is an advancement in the state of the art of: measurement and metrology fundamentals, sensors, measurement instruments, measurement and estimation techniques, measurement data processing and fusion algorithms, evaluation procedures and methodologies for plants and industrial processes, performance analysis of systems, processes and algorithms, mathematical models for measurement-oriented purposes, distributed measurement systems in a connected world.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信