Rui Song , Yingji Li , Mingjie Tian , Hanwen Wang , Fausto Giunchiglia , Hao Xu
{"title":"利用大型语言模型反馈进行因果关键词驱动的可靠文本分类","authors":"Rui Song , Yingji Li , Mingjie Tian , Hanwen Wang , Fausto Giunchiglia , Hao Xu","doi":"10.1016/j.ipm.2024.103964","DOIUrl":null,"url":null,"abstract":"<div><div>Recent studies show Pre-trained Language Models (PLMs) tend to shortcut learning, reducing effectiveness with Out-Of-Distribution (OOD) samples, prompting research on the impact of shortcuts and robust causal features by interpretable methods for text classification. However, current approaches encounter two primary challenges. Firstly, black-box interpretable methods often yield incorrect causal keywords. Secondly, existing methods do not differentiate between shortcuts and causal keywords, often employing a unified approach to deal with them. To address the first challenge, we propose a framework that incorporates Large Language Model’s feedback into the process of identifying shortcuts and causal keywords. Specifically, we transform causal feature extraction into a word-level binary labeling task with the aid of ChatGPT. For the second challenge, we introduce a multi-grained shortcut mitigation framework. This framework includes two auxiliary tasks aimed at addressing shortcuts and causal features separately: shortcut reconstruction and counterfactual contrastive learning. These tasks enhance PLMs at both the token and sample granularity levels, respectively. Experimental results show that the proposed method achieves an average performance improvement of more than 1% under the premise of four different language model as the backbones for sentiment classification and toxicity detection tasks on 8 datasets compared with the most recent baseline methods.</div></div>","PeriodicalId":50365,"journal":{"name":"Information Processing & Management","volume":"62 2","pages":"Article 103964"},"PeriodicalIF":7.4000,"publicationDate":"2024-11-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Causal keyword driven reliable text classification with large language model feedback\",\"authors\":\"Rui Song , Yingji Li , Mingjie Tian , Hanwen Wang , Fausto Giunchiglia , Hao Xu\",\"doi\":\"10.1016/j.ipm.2024.103964\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><div>Recent studies show Pre-trained Language Models (PLMs) tend to shortcut learning, reducing effectiveness with Out-Of-Distribution (OOD) samples, prompting research on the impact of shortcuts and robust causal features by interpretable methods for text classification. However, current approaches encounter two primary challenges. Firstly, black-box interpretable methods often yield incorrect causal keywords. Secondly, existing methods do not differentiate between shortcuts and causal keywords, often employing a unified approach to deal with them. To address the first challenge, we propose a framework that incorporates Large Language Model’s feedback into the process of identifying shortcuts and causal keywords. Specifically, we transform causal feature extraction into a word-level binary labeling task with the aid of ChatGPT. For the second challenge, we introduce a multi-grained shortcut mitigation framework. This framework includes two auxiliary tasks aimed at addressing shortcuts and causal features separately: shortcut reconstruction and counterfactual contrastive learning. These tasks enhance PLMs at both the token and sample granularity levels, respectively. Experimental results show that the proposed method achieves an average performance improvement of more than 1% under the premise of four different language model as the backbones for sentiment classification and toxicity detection tasks on 8 datasets compared with the most recent baseline methods.</div></div>\",\"PeriodicalId\":50365,\"journal\":{\"name\":\"Information Processing & Management\",\"volume\":\"62 2\",\"pages\":\"Article 103964\"},\"PeriodicalIF\":7.4000,\"publicationDate\":\"2024-11-18\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Information Processing & Management\",\"FirstCategoryId\":\"94\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S0306457324003236\",\"RegionNum\":1,\"RegionCategory\":\"管理学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"COMPUTER SCIENCE, INFORMATION SYSTEMS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Information Processing & Management","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0306457324003236","RegionNum":1,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, INFORMATION SYSTEMS","Score":null,"Total":0}
Causal keyword driven reliable text classification with large language model feedback
Recent studies show Pre-trained Language Models (PLMs) tend to shortcut learning, reducing effectiveness with Out-Of-Distribution (OOD) samples, prompting research on the impact of shortcuts and robust causal features by interpretable methods for text classification. However, current approaches encounter two primary challenges. Firstly, black-box interpretable methods often yield incorrect causal keywords. Secondly, existing methods do not differentiate between shortcuts and causal keywords, often employing a unified approach to deal with them. To address the first challenge, we propose a framework that incorporates Large Language Model’s feedback into the process of identifying shortcuts and causal keywords. Specifically, we transform causal feature extraction into a word-level binary labeling task with the aid of ChatGPT. For the second challenge, we introduce a multi-grained shortcut mitigation framework. This framework includes two auxiliary tasks aimed at addressing shortcuts and causal features separately: shortcut reconstruction and counterfactual contrastive learning. These tasks enhance PLMs at both the token and sample granularity levels, respectively. Experimental results show that the proposed method achieves an average performance improvement of more than 1% under the premise of four different language model as the backbones for sentiment classification and toxicity detection tasks on 8 datasets compared with the most recent baseline methods.
期刊介绍:
Information Processing and Management is dedicated to publishing cutting-edge original research at the convergence of computing and information science. Our scope encompasses theory, methods, and applications across various domains, including advertising, business, health, information science, information technology marketing, and social computing.
We aim to cater to the interests of both primary researchers and practitioners by offering an effective platform for the timely dissemination of advanced and topical issues in this interdisciplinary field. The journal places particular emphasis on original research articles, research survey articles, research method articles, and articles addressing critical applications of research. Join us in advancing knowledge and innovation at the intersection of computing and information science.