DTKD-DL: Dual-teacher knowledge distillation with dual-loops for continuous few-shot relation extraction

Abstract

This paper introduces a new model named DTKD-DL, designed to address the issue of Continuous Few-Shot Relation Extraction (CFRE) across tasks, with the goal of learning and adapting to newly emerging relations while reducing catastrophic forgetting. In this paper, we have designed a dual-teacher knowledge distillation model based on relation information to enrich knowledge representation and retain prior knowledge. We employ a dual-loops distillation approach, which facilitates knowledge transfer and optimizes the direction of parameter updates, thereby reducing the occurrence of catastrophic forgetting. Furthermore, to avoid overfitting issues caused by multiple rounds of distillation, we have innovatively integrated reinforcement learning with the model. We have validated our model on the FewRel and TACRED datasets and compared it with the large language model Llama3-8b, demonstrating the effectiveness of our model in this scenario and its advantages over the most advanced methods, achieving state-of-the-art results.