Correlation-Based Knowledge Distillation in Exemplar-Free Class-Incremental Learning

Abstract

Class-incremental learning (CIL) aims to learn a family of classes incrementally with data available in order rather than training all data at once. One main drawback of CIL is that standard deep neural networks suffer from catastrophic forgetting (CF), especially when the model only has access to data from the current incremental step. Knowledge Distillation (KD) is a widely used technique that utilizes old models as the teacher model to alleviate CF. However, based on a case study, our investigation reveals that the vanilla KD is insufficient with a strict point-to-point restriction. Instead, a relaxed match between the teacher and student improves distillation performance and model stability. In this article, we propose a simple yet effective method to mitigate CF without any additional training costs or requiring any exemplars. Specifically, we apply the linear correlation between the features of the teacher and student to measure the distillation loss rather than vanilla point-to-point loss, which significantly improves the model stability. Then, we utilize label augmentation to improve feature generalization and save prototypes to alleviate classification bias further. The proposed method significantly outperforms state-of-the-art methods in the various settings of benchmarks, including CIFAR-100 and Tiny-ImageNet, demonstrating its effectiveness and robustness.