Yun Liu , Xiaoming Zhang , Bo Zhang , Guofeng He , Ke Zhou , Zhoujun Li
{"title":"社会链接下基于标签语义引导的多模态情感分析","authors":"Yun Liu , Xiaoming Zhang , Bo Zhang , Guofeng He , Ke Zhou , Zhoujun Li","doi":"10.1016/j.patcog.2025.112277","DOIUrl":null,"url":null,"abstract":"<div><div>The proliferation of social media platforms has led to an explosion of multimodal data that encapsulates rich emotional content. Effectively integrating heterogeneous modalities to predict sentiment polarity remains a critical challenge. Existing approaches often underexploit sentiment prior knowledge and largely ignore the impact of social links on emotional trends, resulting in suboptimal performance. To address these limitations, we propose a novel multimodal sentiment analysis framework, i.e., Label Semantic Guidance under Social Links (LSGSL). LSGSL enhances sentiment reasoning by jointly modeling visual-textual features and the social relationships between users. Specifically, it encodes social links as a graph structure to facilitate sentiment-aware interactions across modalities, and introduces a novel use of sentiment labels-not merely as classification targets, but as semantic embeddings that guide the fusion and reasoning processes. Furthermore, LSGSL adopts a multi-task learning paradigm that jointly optimizes three objectives: image-text contrastive loss, sentiment-guided semantic similarity loss, and sentiment polarity classification loss. Extensive experiments on three widely-used benchmark datasets demonstrate that LSGSL consistently outperforms state-of-the-art methods, offering new insights into the role of social context and semantic label guidance in multimodal sentiment analysis.</div></div>","PeriodicalId":49713,"journal":{"name":"Pattern Recognition","volume":"171 ","pages":"Article 112277"},"PeriodicalIF":7.6000,"publicationDate":"2025-08-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Multimodal sentiment analysis based on label semantic guidance under social links\",\"authors\":\"Yun Liu , Xiaoming Zhang , Bo Zhang , Guofeng He , Ke Zhou , Zhoujun Li\",\"doi\":\"10.1016/j.patcog.2025.112277\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><div>The proliferation of social media platforms has led to an explosion of multimodal data that encapsulates rich emotional content. Effectively integrating heterogeneous modalities to predict sentiment polarity remains a critical challenge. Existing approaches often underexploit sentiment prior knowledge and largely ignore the impact of social links on emotional trends, resulting in suboptimal performance. To address these limitations, we propose a novel multimodal sentiment analysis framework, i.e., Label Semantic Guidance under Social Links (LSGSL). LSGSL enhances sentiment reasoning by jointly modeling visual-textual features and the social relationships between users. Specifically, it encodes social links as a graph structure to facilitate sentiment-aware interactions across modalities, and introduces a novel use of sentiment labels-not merely as classification targets, but as semantic embeddings that guide the fusion and reasoning processes. Furthermore, LSGSL adopts a multi-task learning paradigm that jointly optimizes three objectives: image-text contrastive loss, sentiment-guided semantic similarity loss, and sentiment polarity classification loss. Extensive experiments on three widely-used benchmark datasets demonstrate that LSGSL consistently outperforms state-of-the-art methods, offering new insights into the role of social context and semantic label guidance in multimodal sentiment analysis.</div></div>\",\"PeriodicalId\":49713,\"journal\":{\"name\":\"Pattern Recognition\",\"volume\":\"171 \",\"pages\":\"Article 112277\"},\"PeriodicalIF\":7.6000,\"publicationDate\":\"2025-08-22\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Pattern Recognition\",\"FirstCategoryId\":\"94\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S0031320325009380\",\"RegionNum\":1,\"RegionCategory\":\"计算机科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Pattern Recognition","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0031320325009380","RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE","Score":null,"Total":0}
Multimodal sentiment analysis based on label semantic guidance under social links
The proliferation of social media platforms has led to an explosion of multimodal data that encapsulates rich emotional content. Effectively integrating heterogeneous modalities to predict sentiment polarity remains a critical challenge. Existing approaches often underexploit sentiment prior knowledge and largely ignore the impact of social links on emotional trends, resulting in suboptimal performance. To address these limitations, we propose a novel multimodal sentiment analysis framework, i.e., Label Semantic Guidance under Social Links (LSGSL). LSGSL enhances sentiment reasoning by jointly modeling visual-textual features and the social relationships between users. Specifically, it encodes social links as a graph structure to facilitate sentiment-aware interactions across modalities, and introduces a novel use of sentiment labels-not merely as classification targets, but as semantic embeddings that guide the fusion and reasoning processes. Furthermore, LSGSL adopts a multi-task learning paradigm that jointly optimizes three objectives: image-text contrastive loss, sentiment-guided semantic similarity loss, and sentiment polarity classification loss. Extensive experiments on three widely-used benchmark datasets demonstrate that LSGSL consistently outperforms state-of-the-art methods, offering new insights into the role of social context and semantic label guidance in multimodal sentiment analysis.
期刊介绍:
The field of Pattern Recognition is both mature and rapidly evolving, playing a crucial role in various related fields such as computer vision, image processing, text analysis, and neural networks. It closely intersects with machine learning and is being applied in emerging areas like biometrics, bioinformatics, multimedia data analysis, and data science. The journal Pattern Recognition, established half a century ago during the early days of computer science, has since grown significantly in scope and influence.