BERT Model Compression With Decoupled Knowledge Distillation And Representation Learning

Abstract

Pre-trained language models such as BERT have proven essential in natural language processing(NLP). However, their huge number of parameters and training cost make them very limited in practical deployment. To overcome BERT’s lack of computing resources, we propose a BERT compression method by applying decoupled knowledge distillation and representation learning, compressing the large model(teacher) into a lightweight network(student). Decoupled knowledge distillation divides the classical distillation loss into target related knowledge distillation(TRKD) and non-target related knowledge distillation(NRKD). Representation learning pools the Transformer output of each two layers, and the student network learns the intermediate features of the teacher network. It has better results on tasks of Sentiment Classification and Paraphrase Similarity Matching, retaining 98.9% performance of the large model.