Supervised contrastive learning with prototype distillation for data incremental learning

Abstract

The goal of Data Incremental Learning (DIL) is to enable learning from small-scale data batches from non-stationary data streams without clear task divisions. A challenge in this domain is the occurrence of catastrophic forgetting in deep neural networks. To effectively address the challenges inherent to DIL, the trained models must exhibit stability and flexibility, ensuring the retention of information from previously learned classes while adapting to incorporate new ones. Prototypes are particularly effective for classifying separable embeddings within the feature space, as they consolidate embeddings from the same class and push those from different classes further apart. This aligns with the principles of contrastive learning. In this paper, we propose Supervised Contrastive learning with the Prototype Distillation (SCPD) method for the DIL problem. First, we employ supervised contrastive loss (SCL) for model training to enhance the class separability of the extracted model representations and improve the flexibility of the model. To further mitigate the forgetting problem, we propose a prototype distillation loss (PDL), ensuring that feature representations remain close to their corresponding prototypes, enhancing the model’s stability. The integration of SCL and PDL within SCPD ensures both the stability and flexibility of the model. Experimental results demonstrate that the SCPD method outperforms prior state-of-the-art approaches across several benchmarks, including those with various imbalanced setups.