Effective Generative Replay with Strong Memory for Continual Learning

Abstract

Continual learning enables artificial neural networks (ANNs) to recognize samples derived from unknown classes while maintaining high classification accuracy for known classes. A classic continual learning approach involves storing data acquired from previously learned tasks and replaying it alongside new tasks in subsequent training sessions. However, data storage may not be feasible due to privacy or security concerns. To address this issue, we propose a effective approach for retaining a strong memory of past tasks within the utilized model. Our method integrates visual saliency-based feature enhancement with a generative replay strategy that captures past task information using visual saliency cues. Specifically, we integrate an adaptive sparse convolutional network module into a generative model, where adaptive sparse convolutional layers select task-relevant features and reduce the number of redundant computations and storage. Experiments show that our method reduces computational overhead by approximately 8% compared to the baseline method. Additionally, since sparse convolution can lead to the loss of global contextual information, we incorporate a bottleneck attention module to improve the feature representations, resulting in an accuracy improvement of the model in the CIFAR-100 task from 26.90% to 27.50%. Finally, to classify unobserved data not included in the training set, we introduce an adaptive mask (AM) module. In the CIFAR-100 20-stage task, the model accuracy improved from 16.05% (ASC only) to 20.31%, and the number of parameter calculations is reduced by 5.1%. This method effectively addresses data retention challenges while enhancing performance and provides a promising solution for privacy-preserving continual learning.