An adaptive knowledge distillation algorithm for text classification

Abstract

Using knowledge distillation to compress pre-trained models such as Bert has proven to be highly effective in text classification tasks. However, the overhead of tuning parameters manually still hinders their application in practice. To alleviate the cost of manual tuning of parameters in training tasks, inspired by the inverse decrease of the word frequency of TF-IDF, this paper proposes an adaptive knowledge distillation method (AKD). This core idea of the method is based on the Cosine similarity score which is calculated by the probabilistic outputs similarity measurement in two networks. The higher the score, the closer the student model's understanding of knowledge is to the teacher model, and the lower the degree of imitation of the teacher model. On the contrary, we need to increase the degree to which the student model imitates the teacher model. Interestingly, this method can improve distillation model quality. Experimental results show that the proposed method significantly improves the precision, recall and F1 value of text classification tasks. However, training speed of AKD is slightly slower than baseline models. This study provides new insights into knowledge distillation.