Knowledge Distillation by Multiple Student Instance Interaction

Abstract

Knowledge distillation is an efficient method in neural network compression, which transfers the knowledge from a high-capacity teacher network to a low-capacity student network. Previous approaches follow the ‘one teacher and one student’ paradigm, which neglects the possibility that interaction of multiple students could boost the distillation performance. In this paper, we propose a novel approach by simultaneously training multiple instances of a student model. By adding the similarity and diversity losses into the baseline knowledge distillation and adaptively adjusting the proportion of these losses according to accuracy changes of multiple student instances, we build a distillation system to make students collaborate and compete with each other, which improves system robustness and performance. Experiments show superior performance of the proposed method over existing offline and online distillation schemes on datasets with various scales.