Automatic Optimization of super Parameters Based on Model Pruning and Knowledge Distillation

Abstract

In recent years, deep neural network has been widely used in computer vision, speech recognition and other fields. However, to obtain better performance, it needs to design a network with higher complexity, and the corresponding model calculation amount and storage space are also increasing. At the same time, the computing resources and energy consumption budget of mobile devices are very limited. Therefore, model compression is very important for deploying neural network models on mobile devices. Knowledge distillation technology based on transfer learning is an effective method to realize model compression. This study proposes: the model pruning technology is introduced into the student network design of knowledge distillation, and the super parameters (temperature T, scale factor λ, pruning rate ϒ) are automatically optimized, and the optimal combination of parameters is selected as the final value according to the final performance. The results show that, compared with the commonly used pruning techniques, this method can effectively improve the accuracy of the network without increasing the network size, and the network performance can be further improved by adjusting the value of super parameters.