Continual Semantic Segmentation With Tiny Memory

Abstract

Deep neural networks suffer from catastrophic forgetting when sequentially training on image semantic segmentation tasks. Storing historical images and utilizing them to train the model alongside data of new tasks can alleviate catastrophic forgetting but result in a large memory cost and an increase in training time, which limits practical applications. In this paper, we propose a continual semantic segmentation framework with tiny memory to address the catastrophic forgetting of continual semantic segmentation. To reduce memory utilization, we store small patches of specific semantic regions instead of storing entire images. This strategy eliminates the large amount of image backgrounds, allowing for the storage of a larger number of trimmed samples. Furthermore, we introduce a diffusion-based module to generate augmented images with various backgrounds to enhance the diversity of the trimmed sample images. Equipped with our dynamic similarity loss, our method achieves favourable performance on two widely used benchmarks, Pascal-VOC 2012 and ADE 20K, compared to several state-of-the-art methods while significantly reducing storage space usage.