Pruning for efficient DenseNet via surrogate-model-assisted genetic algorithm considering neural architecture search proxies

Abstract

Recently, convolution neural networks have achieved remarkable progress in computer vision. These neural networks have a large number of parameters, which should be limited in resource-constrained environments. To address this problem, new pruning approaches have explored using neural architecture search (NAS) to determine optimal subnetworks. We propose a novel pruning framework using a surrogate model-assisted genetic algorithm considering NAS proxies (SMA-GA-NP). We applied multi-dimensional encoding and designed crossover and mutation methods. To reduce the search time of NAS, we leveraged a surrogate model to approximate the fitness value of individuals and used NAS proxies, such as reducing the number of epochs and the training set size. The DenseNet-BC (k $=$ 12) model was used as the baseline. We achieved highly competitive performance on CIFAR-10 compared with other GA-based pruning methods and baselines. For CIFAR-100, we reduced the number of parameters by 11.25% to 18.75%, while achieving less than 1% performance degradation compared to the baseline model. These findings highlight SMA-GA-NP’s effectiveness in significantly reducing the number of parameters while having a negligible impact on the model’s performance. We also conducted an ablation study to explore the efficiency of the GA settings, the surrogate model, and NAS proxies in SMA-GA-NP and identified the current limitations and future potential of SMA-GA-NP.