Puning Zhang;Miao Fu;Rongjian Zhao;Hongbin Zhang;Changchun Luo
{"title":"PURE: Personality-Coupled Multi-Task Learning Framework for Aspect-Based Multimodal Sentiment Analysis","authors":"Puning Zhang;Miao Fu;Rongjian Zhao;Hongbin Zhang;Changchun Luo","doi":"10.1109/TKDE.2024.3485108","DOIUrl":null,"url":null,"abstract":"Aspect-Based Multimodal Sentiment Analysis (ABMSA) aims to infer the users’ sentiment polarities over individual aspects using visual, textual, and acoustic signals. Although psychological studies have shown that personality has a direct impact on people's sentiment orientations, most existing methods disregard the potential personality character while executing ABMSA tasks. To tackle this issue, a novel psychological perspective, the people's personalities are introduced. To the best of our knowledge, this paper is the very first study in this field. Different from current pipelined multi-task sentiment analysis methods, an end-to-end ABMSA method called Personality-coupled mUlti-task leaRning framEwork (PURE) is proposed, which strongly couples personality mining and ABMSA tasks in a unified architecture to avoid error propagation and enhance the overall system robustness. Specifically, an adaptive personality feature extraction method is designed to accurately model the first impression of different people's personalities. Then, a multi-task ABMSA framework is designed to strongly couple the multimodal features of aspects extracted by the interactive attention fusion network with people's personalities. Subsequently, the proposed framework optimizes them parallel via extended Bayesian meta-learning. Finally, compared to the current optimal model, the classification accuracy and macro F1 score of the proposed model have both shown significant improvements on public datasets. In addition, PURE is transferable and can effectively couple personality modeling tasks with any other sentiment analysis methods.","PeriodicalId":13496,"journal":{"name":"IEEE Transactions on Knowledge and Data Engineering","volume":"37 1","pages":"462-477"},"PeriodicalIF":8.9000,"publicationDate":"2024-10-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Transactions on Knowledge and Data Engineering","FirstCategoryId":"94","ListUrlMain":"https://ieeexplore.ieee.org/document/10731889/","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE","Score":null,"Total":0}
引用次数: 0
Abstract
Aspect-Based Multimodal Sentiment Analysis (ABMSA) aims to infer the users’ sentiment polarities over individual aspects using visual, textual, and acoustic signals. Although psychological studies have shown that personality has a direct impact on people's sentiment orientations, most existing methods disregard the potential personality character while executing ABMSA tasks. To tackle this issue, a novel psychological perspective, the people's personalities are introduced. To the best of our knowledge, this paper is the very first study in this field. Different from current pipelined multi-task sentiment analysis methods, an end-to-end ABMSA method called Personality-coupled mUlti-task leaRning framEwork (PURE) is proposed, which strongly couples personality mining and ABMSA tasks in a unified architecture to avoid error propagation and enhance the overall system robustness. Specifically, an adaptive personality feature extraction method is designed to accurately model the first impression of different people's personalities. Then, a multi-task ABMSA framework is designed to strongly couple the multimodal features of aspects extracted by the interactive attention fusion network with people's personalities. Subsequently, the proposed framework optimizes them parallel via extended Bayesian meta-learning. Finally, compared to the current optimal model, the classification accuracy and macro F1 score of the proposed model have both shown significant improvements on public datasets. In addition, PURE is transferable and can effectively couple personality modeling tasks with any other sentiment analysis methods.
期刊介绍:
The IEEE Transactions on Knowledge and Data Engineering encompasses knowledge and data engineering aspects within computer science, artificial intelligence, electrical engineering, computer engineering, and related fields. It provides an interdisciplinary platform for disseminating new developments in knowledge and data engineering and explores the practicality of these concepts in both hardware and software. Specific areas covered include knowledge-based and expert systems, AI techniques for knowledge and data management, tools, and methodologies, distributed processing, real-time systems, architectures, data management practices, database design, query languages, security, fault tolerance, statistical databases, algorithms, performance evaluation, and applications.