Invariant prompting with classifier rectification for continual learning

Continual learning aims to train a model capable of continuously learning and retaining knowledge from a sequence of tasks. Recently, prompt-based continual learning has been proposed to leverage the generalization ability of a pre-trained model with task-specific prompts for instruction. Prompt component training is a promising approach to enhancing the plasticity for prompt-based continual learning. Nevertheless, this approach changes the instructed features to be noisy for query samples from the old tasks. Additionally, the problem of scale misalignment in classifier logits between different tasks leads to misclassification. To address these issues, we propose an invariant Prompting with Classifier Rectification (iPrompt-CR) method for prompt-based continual learning. In our method, the learnable keys corresponding to each new-task component are constrained to be orthogonal to the query prototype in the old tasks for invariant prompting, which reduces feature representation noise. After prompt learning, instructed features are sampled from Gaussian-distributed prototypes for classifier rectification with unified logit scale for more accurate predictions. Extensive experimental results on four benchmark datasets demonstrate that our method outperforms the state of the arts in both class-incremental learning and more realistic general incremental learning scenarios.