Adaptive Logit Reconstruction in Knowledge Distillation

Abstract

In the logit-based knowledge distillation method, the student model learns the classification information of the teacher network by transmitting high-dimensional and abstract logits. Nevertheless, the teacher network is not an optimal learning target. On common datasets such as CIFAR100 and ImageNet, the majority of models exhibit classification accuracies of only 60% to 80%. These errors in the teacher models are a significant part of knowledge distillation that cannot be ignored. In order to facilitate the acquisition of more accurate knowledge by students, we propose the implementation of adaptive logit reconstruction knowledge distillation (ALRKD). ALRKD corrects errors by using the standard deviation, which represents the fluctuation degree of the logit distribution. Furthermore, in order to compensate for the loss of information that occurs during the correction process, an additional branch is designed to provide supplementary knowledge regarding the relationships between other classes. The results of several experiments on common datasets demonstrate the significant superiority of ALRKD.