CausalEnhance: knowledge-enhanced pre-training for causality identification and extraction

Abstract

Causality identification and extraction are crucial in understanding causal relationships in text. Current studies heavily rely on datasets annotated with causal relationships. However, acquiring such datasets poses a challenge due to substantial costs, hindering progress in this research field. To address this, we introduce CausalEnhance, a novel approach designed to bridge this gap by combining weakly-guided pre-training with external causal knowledge. Our method starts with a rule-based system that automates causal annotation, enriching external data with explicit causal knowledge and creating pseudo labels. These pseudo-labels are then incorporated into a weakly supervised pre-training framework. We introduce three innovative pre-training tasks: the Pre-training Causal Clues Fill-Mask task (PCM) to pinpoint causality origins, the Pre-training Causality Identification task (PCI) to capture general causal patterns, and the Pre-training Causality Extraction task (PCE) for understanding explicit causal pairs and inferring implicit ones. Our experiments, conducted across eight datasets in two languages, English and Chinese, demonstrate CausalEnhance’s effectiveness in both identifying and extracting causality, highlighting its potential as a robust method for textual causality analysis in different linguistic contexts.