Dynamic network compression via probabilistic channel pruning

Neural network compression problems have been extensively studied to overcome the limitations of compute-intensive deep learning models. Most of the state-of-the-art solutions in this context are based on network pruning that identify and remove unimportant weights, filters or channels. However, existing methods often lack actual speedup or require complex pruning criteria and additional training (fine-tuning) overhead. To address these limitations, we develop probability-based connectivity module that determines the connection of each channel to the next layer. Our connectivity module enables to dynamically activate and deactivate channel connections during training, and hence, does not necessitate fine-tuning of the pruned model. We show that the convolution decomposition, which decomposes convolution with connectivity module and depth-wise convolution can effectively induce sparsity, resulting in 52.76 %, 46.05 % reduction of parameter counts, with even boosting accuracy (+0.19 %, + 0.3 %) compared to baseline architectures in ResNet-56, VGG-19 Models. We also introduce resource-aware regularization that exploits the probabilistic activation of connectivity module in order to control the level of compression. We show that our method achieves comparable level of compression and accuracy to the state-of-the-art pruning methods.