Two-dimensional normalized knowledge distillation leveraging class relations

Knowledge distillation (KD) as one of the important methods of model compression, has been widely used in tasks such as image classification and detection. Existing KD methods are mainly carried out at the instance level and often ignore the role of inter-class relational information. Additionally, when there is a significant gap between the student’s capacity and the teacher’s capacity, the two model cannot be matched precisely. To address these issues, this paper proposes a two-dimensional normalized knowledge distillation method, which integrates inter-class and intra-class correlations and rectifies logits in two dimensions. Through our approach, the student model is able to acquire contextual information between samples with the help of intra-class correlation and mitigate the effect of logits magnitude on the prediction results through normalized rectification. We conduct numerous experiments and results show that our method achieves higher accuracy and better training efficiency compared to traditional KD methods.