Toward Generalized Entropic Sparsification for Convolutional Neural Networks.

Abstract

Convolutional neural networks (CNNs) are reported to be overparametrized. The search for optimal (minimal) and sufficient architecture is an NP-hard problem: if the network has $N$ neurons, then there are 2$^{N}$ possibilities to connect them-and therefore 2$^{N}$ possible architectures and 2$^{N}$ Boolean hyperparameters to encode them. Selecting the best possible hyperparameter out of them becomes an $N^{p}$ -hard problem since 2$^{N}$ grows in $N$ faster then any polynomial $N^{p}$. Here, we introduce a layer-by-layer data-driven pruning method based on the mathematical idea aiming at a computationally scalable entropic relaxation of the pruning problem. The sparse subnetwork is found from the pretrained (full) CNN using the network entropy minimization as a sparsity constraint. This allows deploying a numerically scalable algorithm with a sublinear scaling cost. The method is validated on several benchmarks (architectures): on MNIST (LeNet), resulting in sparsity of 55% to 84% and loss in accuracy of just 0.1% to 0.5%, and on CIFAR-10 (VGG-16, ResNet18), resulting in sparsity of 73% to 89% and loss in accuracy of 0.1% to 0.5%.