Reverse Self-Distillation Overcoming the Self-Distillation Barrier

Abstract

Deep neural networks generally cannot gather more helpful information with limited data in image classification, resulting in poor performance. Self-distillation, as a novel knowledge distillation technique, integrates the roles of teacher and student into a single network to solve this problem. A better understanding of the efficiency of self-distillation is critical to its advancement. In this article, we provide a new perspective: the effectiveness of self-distillation comes not only from distillation but also from the supervisory information provided by the shallow networks. At the same time, we find a barrier that limits the effectiveness of self-distillation. Based on this, reverse self-distillation is proposed. In contrast to self-distillation, the internal knowledge flow is in the opposite direction. Experimental results show that reverse self-distillation can break the barrier of self-distillation and further improve the accuracy of networks. On average, 2.8% and 3.2% accuracy boosts are observed on CIFAR100 and TinyImageNet.