High Efficient Compression: Model Compression Method Based on Channel Pruning and Knowledge Distillation

Abstract

With the development of neural network models in the field of deep learning image processing, more complex and higher performance neural network models can be used, but with it comes greater parameter quantities and computing power requirements. Although the model compression method has been updated in recent years, it still has problems such as inefficiency and reduced accuracy. Based on this, we propose a model compression method based on channel pruning and knowledge distillation, we sort and prune the L2 regularization weights of the convolutional layer filter containing a large number of calculations in the original model and prune it according to a certain proportion, and assist in restoring the accuracy by the knowledge distillation of the pruning model through the original model. Experimental results of the VGG16 model show that our pruning method reduced the parameters of the model by half and increased its processing speed to 2.7 times that of the original model, but the accuracy was reduced by about 10%. To solve this problem, we propose a knowledge distillation method to assist the training of the pruning model, so that the problem of model accuracy degradation has been improved and maintained at about 3.3%, to achieve a balanced state of model compression and accuracy reduction, thereby realizing the efficient compression of the model.