{"title":"GAP: A group-based automatic pruning algorithm via convolution kernel fusion","authors":"","doi":"10.1016/j.neucom.2024.128488","DOIUrl":null,"url":null,"abstract":"<div><p>In recent years, the deployment and operation of convolution neural networks on edge devices with limited computing capabilities have become increasingly challenging due to the large network structure and computational cost. Currently, the mainstream structured pruning algorithms mainly compress the network at the filter or layer level. However, these methods introduce too much human intervention with large granularities, which may lead to unpredictable performance after compression. In this paper, we propose a group-based automatic pruning algorithm(GAP) via kernel fusion to automatically search for the optimal pruning structure in a more fine-grained manner. Specifically, we first adopt a novel nonlinear dimensionality reduction clustering algorithm to divide the filters of each convolution layer into groups of equal size. Afterwards, we encode the mutual distribution similarity of the kernels within each group, and its KL divergence is employed as an importance indicator to determine the retained kernel groups through weighted fusion. Subsequently, we introduce an intelligent searching module that automatically explore and optimize the pruned structure of each layer. Finally, the pruned filters are permutated to form a dense group convolution and fine-tuned. Sufficient experiments show that, on two image classification datasets, for five advanced CNN models, our GAP algorithm outperforms most extant SOTA schemes, reduces artificial intervention, and enables efficient end-to-end training of compact models.</p></div>","PeriodicalId":19268,"journal":{"name":"Neurocomputing","volume":null,"pages":null},"PeriodicalIF":5.5000,"publicationDate":"2024-08-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Neurocomputing","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0925231224012591","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE","Score":null,"Total":0}
引用次数: 0
Abstract
In recent years, the deployment and operation of convolution neural networks on edge devices with limited computing capabilities have become increasingly challenging due to the large network structure and computational cost. Currently, the mainstream structured pruning algorithms mainly compress the network at the filter or layer level. However, these methods introduce too much human intervention with large granularities, which may lead to unpredictable performance after compression. In this paper, we propose a group-based automatic pruning algorithm(GAP) via kernel fusion to automatically search for the optimal pruning structure in a more fine-grained manner. Specifically, we first adopt a novel nonlinear dimensionality reduction clustering algorithm to divide the filters of each convolution layer into groups of equal size. Afterwards, we encode the mutual distribution similarity of the kernels within each group, and its KL divergence is employed as an importance indicator to determine the retained kernel groups through weighted fusion. Subsequently, we introduce an intelligent searching module that automatically explore and optimize the pruned structure of each layer. Finally, the pruned filters are permutated to form a dense group convolution and fine-tuned. Sufficient experiments show that, on two image classification datasets, for five advanced CNN models, our GAP algorithm outperforms most extant SOTA schemes, reduces artificial intervention, and enables efficient end-to-end training of compact models.
近年来,由于网络结构庞大、计算成本高昂,在计算能力有限的边缘设备上部署和运行卷积神经网络变得越来越具有挑战性。目前,主流的结构剪枝算法主要是在滤波器或层级别上压缩网络。然而,这些方法引入了过多的人工干预,粒度过大,可能导致压缩后的性能难以预测。在本文中,我们通过内核融合提出了一种基于组的自动剪枝算法(GAP),以更精细的方式自动搜索最佳剪枝结构。具体来说,我们首先采用一种新颖的非线性降维聚类算法,将每个卷积层的滤波器分成大小相等的组。然后,我们对每个组内内核的相互分布相似性进行编码,并将其 KL 发散作为重要性指标,通过加权融合确定保留的内核组。随后,我们引入一个智能搜索模块,自动探索和优化每一层的剪枝结构。最后,对剪枝后的滤波器进行排列组合,形成密集组卷积并进行微调。充分的实验表明,在两个图像分类数据集上,对于五种高级 CNN 模型,我们的 GAP 算法优于大多数现有的 SOTA 方案,减少了人工干预,并实现了紧凑模型的高效端到端训练。
期刊介绍:
Neurocomputing publishes articles describing recent fundamental contributions in the field of neurocomputing. Neurocomputing theory, practice and applications are the essential topics being covered.