Overcome STSF phenomenon in catastrophic forgetting

Abstract

Catastrophic forgetting is an undesirable phenomenon in convolution neural networks and iCarl is an effective algorithm for preventing catastrophic forgetting. However, a potential defect that more similar tasks result in severer catastrophic forgetting (STSF) in iCarl is explored in this work. The reason of STSF is that similar tasks are prone to occupy similar feature channels and similar feature representations, thus they can be replaced by each other easily. Based on these findings, a novel method Similar Margin Loss (SML) is proposed. SML aims to make feature representations of samples from the same task compact while making feature representations from the different tasks differentiable in the feature space. Experiment results show that SML is effective in alleviating STSF.