Preserve Knowledge with Auxiliary Feature Extractor for Class Incremental Learning

Abstract

Class incremental learning (CIL) aims to achieve the ability to learn knowledge from the data of novel classes that arrive incrementally. To this end, the exemplar-based method stores a small number of samples of old classes and has been proven to be effective yet it causes the severe data imbalance issue. An approach named SS-IL solves the issue effectively and achieves strong state-of-the-art on large-scale CIL benchmark datasets while behaving badly on small ones. In this paper, we observe that the poor performance of SS-IL on small datasets could stem from not fully stimulating the potentiality of the learned representation of old classes, especially the initial classes. We propose an auxiliary Weight Scaling Feature Extractor (aWSFE) to better maintain and exploit the essential semantics of old classes. This auxiliary extractor is used as a plug-in module with the main classification network based on SS-IL in parallel. We perform a special design for the two branches so that the feature vectors from the main and auxiliary extractor can be integrated easily without an additional aggregation process. After obtaining the updated representations, we finetuning the classifier based on a balanced subset of training data to further promote performance. We conduct extensive experiments on two small-scale CIL benchmark datasets: CIFAR-100 and ImageNet-Sub. Results show that the proposed method effectively alleviates the forgetting of old knowledge and significantly improves the performance of SS-IL on small datasets.