Class-wise knowledge distillation

Abstract

Knowledge distillation (KD) transfers knowledge of a teacher model to improve the performance of a student model which is usually equipped with a lower capacity. The standard KD framework, however, neglects that the DNNs exhibit a wide range of class-wise accuracy and the performance of some classes is even decreased after distillation. Observing the above phenomena, we propose a novel Class-Wise Knowledge Distillation method to find the hard classes with a simple yet effective technique and then make the students take more effort to learn these hard classes. In the experiments on image classification tasks using CIFAR-100 dataset, we demonstrate that the proposed method outperforms the other KD methods and achieves excellent performance enhancement on various networks.