{"title":"Correlation mining of multimodal features based on higher-order partial least squares for emotion recognition in conversations","authors":"","doi":"10.1016/j.engappai.2024.109350","DOIUrl":null,"url":null,"abstract":"<div><p>In fields requiring an understanding of emotions, such as digital human interaction and public opinion analysis, achieving a dependable and interpretable model for mining correlations among multimodal features remains a primary objective. However, current deep learning methods often lack transparency and suffer from low interpretability. To address these challenges, we propose a novel Correlation Mining method based on Higher-Order Partial Least Squares (HOPLS) for multimodal Emotion Recognition in conversations (CMHER) in this paper. CMHER innovatively combines HOPLS with transformers and Gated Recurrent Units (GRUs) to compute correlation matrices within unimodal data streams and between cross-modal sources. HOPLS projects source data into a latent space to predict target data via correlation matrix computations, eliminating the need for Graphical Processing Unit (GPU) acceleration and making it suitable for experimental and edge systems. The integration of HOPLS with deep neural networks involves preprocessing multimodal features into suitable dimensions and latent representations, followed by HOPLS computing correlation matrices for cross-modal latent vectors and final labels through optimal joint subspace approximation, which aims at the improvements of both interpretability and reliability. Additionally, a generalization error fitting module further refines the predicted correlation matrices to improve predictive capability and overall model performance. Experiments on two public datasets validate the superiority of our proposed method.</p></div>","PeriodicalId":50523,"journal":{"name":"Engineering Applications of Artificial Intelligence","volume":null,"pages":null},"PeriodicalIF":7.5000,"publicationDate":"2024-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Engineering Applications of Artificial Intelligence","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0952197624015082","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"AUTOMATION & CONTROL SYSTEMS","Score":null,"Total":0}
引用次数: 0
Abstract
In fields requiring an understanding of emotions, such as digital human interaction and public opinion analysis, achieving a dependable and interpretable model for mining correlations among multimodal features remains a primary objective. However, current deep learning methods often lack transparency and suffer from low interpretability. To address these challenges, we propose a novel Correlation Mining method based on Higher-Order Partial Least Squares (HOPLS) for multimodal Emotion Recognition in conversations (CMHER) in this paper. CMHER innovatively combines HOPLS with transformers and Gated Recurrent Units (GRUs) to compute correlation matrices within unimodal data streams and between cross-modal sources. HOPLS projects source data into a latent space to predict target data via correlation matrix computations, eliminating the need for Graphical Processing Unit (GPU) acceleration and making it suitable for experimental and edge systems. The integration of HOPLS with deep neural networks involves preprocessing multimodal features into suitable dimensions and latent representations, followed by HOPLS computing correlation matrices for cross-modal latent vectors and final labels through optimal joint subspace approximation, which aims at the improvements of both interpretability and reliability. Additionally, a generalization error fitting module further refines the predicted correlation matrices to improve predictive capability and overall model performance. Experiments on two public datasets validate the superiority of our proposed method.
期刊介绍:
Artificial Intelligence (AI) is pivotal in driving the fourth industrial revolution, witnessing remarkable advancements across various machine learning methodologies. AI techniques have become indispensable tools for practicing engineers, enabling them to tackle previously insurmountable challenges. Engineering Applications of Artificial Intelligence serves as a global platform for the swift dissemination of research elucidating the practical application of AI methods across all engineering disciplines. Submitted papers are expected to present novel aspects of AI utilized in real-world engineering applications, validated using publicly available datasets to ensure the replicability of research outcomes. Join us in exploring the transformative potential of AI in engineering.