Functionality Separation: Rethinking Dual-Stream Networks for Class-Incremental Learning

Abstract

Class-incremental learning (CIL) aims to enable models to continually learn new classes without forgetting previously learned ones, often struggling with the stability-plasticity dilemma. The dual-stream architectures adopt one stream with limited learning ability for storing the learned knowledge, and the other stream with high plasticity for learning new information, achieving success in CIL. However, different methods involve completely different components and the design principles for dual-stream architectures remain unclear. In this paper, in contrast to the previous methods that focus on a specific point of dual-stream network design, we conduct a large number of experiments across the board to figure out the principles. It is found that the key to designing dual-stream methods is making each component complete a specific function, namely functionality separation, instead of placing hope in one component to accomplish multiple functions as in many previous works. These crucial functions for dual-stream networks are the widely studied stability-maintaining and plasticity-enhancing, together with the generally overlooked knowledge integration introduced in this work. Additionally, despite a large number of investigations on regularization methods, a simple combination of classification loss and distillation loss is sufficient for the regularization of dual-stream networks. Based on all these experiments and analyses, it is natural to acquire the dual-stream model combined with the compression training paradigm, which effectively separates the functions of stability, plasticity, and the integration of new and old knowledge, achieving the state-of-the-art results on CIFAR-100 and ImageNet-Subset CIL benchmarks. Code is available at https://github.com/gaoqingqing77/DualFS