A soft-masking continual pre-training method based on domain knowledge relevance

Abstract

Most existing continual learning (CL) methods primarily focus on reducing catastrophic forgetting. Although some approaches have achieved CF-free learning, they often treat parameter optimization across tasks as a conflicting process. This assumption not only inhibits the learning of new tasks but also limits knowledge transfer between tasks. To address this, we propose a soft-masking continual pre-training method based on domain knowledge relevance (DKR) within the framework of continual domain-adaptive pre-training. Unlike traditional methods, DKR does not fully regard parameter optimization as adversarial. Instead, it dynamically applied soft masks to model parameters based on the degree of relevance between domain knowledge. Moreover, we introduce an efficient importance calculation method based on dual interactions between input and weights, which accurately assesses the importance of parameters. Our experimental results on six public datasets demonstrate that our method significantly outperforms existing methods in terms of mitigating forgetting. Specifically, DKR achieves negative forgetting with improvements of 1.95% in macro-F1 and 1.52% in accuracy over Naive CL(NCL).