Neural Network Model Pruning without Additional Computation and Structure Requirements

Abstract

In past work, deep learning researchers always designed hyperparameters such as model structure and learning rate first and then used the training set to train the weights in this model. While unrestricted model structure design leads to massive neuron redundancy in neural network models. By pruning these redundant neurons, not only can the storage be compressed effectively, but also the operation can be accelerated. In this paper, we propose a method to utilize the training set to prune the model structure during training: 1) train the initialized model and bring it to basic convergence; 2) feed the entire training set into the model and calculate the activations of neurons in each layer; 3) calculate the threshold for neuron pruning in each layer according to the pruning ratio, delete neurons whose activation value is lower than the threshold, and correspondingly delete the weights of the upper and lower layers; 4) further train the pruned model so that it eventually converges. This method of deleting redundant neurons not only greatly deletes the parameters in the model but also achieves model acceleration. We applied this method to some mainstream neural network models: VGGNet and ResNet, and achieved good results.