Enhancing task incremental continual learning: integrating prompt-based feature selection with pre-trained vision-language model

Abstract

Task incremental continual learning is pivotal for the evolution of general artificial intelligence, enabling models to progressively acquire and integrate new knowledge. However, enhancing model plasticity while ensuring stability remains one of the most significant challenges in this field. In this study, we propose a task incremental continual learning method based on a vision-language model (TICL-VLM), which exhibits both high plasticity and good stability. First, the image encoder from a pre-trained vision-language model (VLM) is adopted for robust feature extraction, and a novel task-prompt-based feature selection module is designed to enhance the plasticity of the proposed model. Additionally, a class description constraint is introduced to further improve the performance of the method. To ensure excellent stability, we freeze the parameters of the VLM's image and text encoders and introduce distinct feature selection and classification modules for each incremental task. Furthermore, a specific dataset (LFDDE) is constructed to comprehensively evaluate the performance of task incremental continual learning algorithms. Extensive experiments have been conducted on both the LFDDE and the well-known CIFAR-100 datasets. The experimental results clearly demonstrate significant improvements in maintaining stability while efficiently incorporating new knowledge with our method.