Causal keyword driven reliable text classification with large language model feedback

Abstract

Recent studies show Pre-trained Language Models (PLMs) tend to shortcut learning, reducing effectiveness with Out-Of-Distribution (OOD) samples, prompting research on the impact of shortcuts and robust causal features by interpretable methods for text classification. However, current approaches encounter two primary challenges. Firstly, black-box interpretable methods often yield incorrect causal keywords. Secondly, existing methods do not differentiate between shortcuts and causal keywords, often employing a unified approach to deal with them. To address the first challenge, we propose a framework that incorporates Large Language Model’s feedback into the process of identifying shortcuts and causal keywords. Specifically, we transform causal feature extraction into a word-level binary labeling task with the aid of ChatGPT. For the second challenge, we introduce a multi-grained shortcut mitigation framework. This framework includes two auxiliary tasks aimed at addressing shortcuts and causal features separately: shortcut reconstruction and counterfactual contrastive learning. These tasks enhance PLMs at both the token and sample granularity levels, respectively. Experimental results show that the proposed method achieves an average performance improvement of more than 1% under the premise of four different language model as the backbones for sentiment classification and toxicity detection tasks on 8 datasets compared with the most recent baseline methods.