基于跨模块融合的多模态情感分析

计算机科学 Pub Date : 2023-02-20 DOI:10.55375/cps.2023.3.2

敬伟张, 彦松李

引用次数: 0

Abstract

理解表达的情感和情绪是多模态情感分析的两个关键因素。人类语言通常是多模态的，包括视觉觉，语音以及文本三个模态，而每个模态又包含众多不同信息，比如文本模态包括基本的语言符号、句法和语言动作等，语音模态包括：语音、语调以及声音表达等。视觉模态包括姿态特征、身体语言、眼神以及面部表达等信息。因此如何高效融合模态间信息便成为当下多模态情感分析领域的一个热点问题。为此，文章提出一种基于跨模块融合网络模型。该模型利用LSTM网络作为语言、视觉模态的表示子网络，同时利用改进升级的Transformer模型的跨模块融合对两种模态信息进行有效融合；为了验证文章中提出的模型的效果，在IEMOCAP和MOSEI数据集上进行了仔细评估，结果表明，该模型针对情感分类的准确度有所提高。

查看原文本刊更多论文

基于跨模块融合的多模态情感分析

Understanding the emotions and emotions expressed are two key factors in multimodal emotional analysis. Human language is usually multimodal, including three modes: visual perception, speech, and text, and each mode contains numerous different information. For example, text mode includes basic language symbols, syntax, and language actions, while speech mode includes speech, intonation, and voice expression. Visual modalities include information such as posture features, body language, eye contact, and facial expression. Therefore, how to efficiently integrate inter modal information has become a hot topic in the field of multimodal emotion analysis. Therefore, the article proposes a cross module fusion network model. This model utilizes the LSTM network as the representation sub network for language and visual modalities, while utilizing the cross module fusion of the improved and upgraded Transformer model to effectively fuse the two modal information; In order to verify the effectiveness of the model proposed in the article, careful evaluation was conducted on the IEMOCAP and MOSEI datasets, and the results showed that the accuracy of the model for sentiment classification has been improved.

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

计算机科学

自引率

0.00%

发文量

21143

期刊介绍： Computer Science (CS) was established in 1974, its original title was Computer Applications and Applied Mathematics until 1979. It is sponsored by Chongqing Southwest Information Co., Ltd, and is the member Journal of CCF（China Computer Federation) and the CCF B Class journal. Computer Science (CS) mainly reports the dynamic development, methodologies and techniques involving a wide range, and International advanced research productions of computer science and technology. Computer Science (CS) has been included in many important national and international index databases, such as CSCD,GCJC, CSA ,DOAJ, IC , UPD, JST. Readers of Computer Science (CS) include the students of college, researches and technicists engaged in the field of computer science and technology