Determining optimal channel partition for 2:4 fine grained structured sparsity

IF 1.3 4区数学 Q2 MATHEMATICS, APPLIED

Optimization Letters Pub Date : 2024-01-11 DOI:10.1007/s11590-023-02084-8

引用次数: 0

Abstract

Deep Neural Networks (DNNs) have demonstrated tremendous success in many applications, but incur high computational burden on the inference side. The 2:4 sparsity pruning method has recently been developed to effectively compress and accelerate DNNs with little to no loss in performance. The method comprises a training phase followed by a pruning step where 2 out of 4 consecutive weights are eliminated to obtain a pruned matrix, which is then retrained to fine-tune the remaining weights. The accuracy of the resultant sparse network is maximized by permuting the matrix along the channel dimension in a way that maximizes the total magnitude of weights preserved during pruning. While earlier works have proposed heuristic methods to generate good permutations, we formalized the problem as a discrete optimization problem. In this paper, we propose four different mathematical programs to determine the optimal permutations and compare their performance for small-sized instances using a standard solver. Further, we develop a complementary column generation scheme to solve DNNs with realistic number of channels.

查看原文本刊更多论文

确定 2:4 细粒度结构稀疏性的最佳信道分区

摘要深度神经网络（DNN）在许多应用中都取得了巨大成功，但在推理方面却产生了很高的计算负担。最近开发的 2:4 稀疏剪枝法能有效压缩和加速 DNN，而且性能几乎没有损失。该方法包括一个训练阶段和一个剪枝步骤，在剪枝步骤中，4 个连续权重中的 2 个被去除，从而得到一个剪枝矩阵，然后对该矩阵进行再训练，以微调剩余权重。通过沿信道维度对矩阵进行排列，使剪枝过程中保留的权重总大小最大化，从而最大限度地提高生成的稀疏网络的准确性。早期的研究提出了启发式方法来生成良好的排列，而我们则将这一问题形式化为离散优化问题。在本文中，我们提出了四种不同的数学方案来确定最优排列，并使用标准求解器对它们在小规模实例中的性能进行了比较。此外，我们还开发了一种补充列生成方案，以解决具有实际通道数的 DNN 问题。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Optimization Letters 管理科学-应用数学

CiteScore

3.40

自引率

6.20%

发文量

116

审稿时长

9 months

期刊介绍： Optimization Letters is an international journal covering all aspects of optimization, including theory, algorithms, computational studies, and applications, and providing an outlet for rapid publication of short communications in the field. Originality, significance, quality and clarity are the essential criteria for choosing the material to be published. Optimization Letters has been expanding in all directions at an astonishing rate during the last few decades. New algorithmic and theoretical techniques have been developed, the diffusion into other disciplines has proceeded at a rapid pace, and our knowledge of all aspects of the field has grown even more profound. At the same time one of the most striking trends in optimization is the constantly increasing interdisciplinary nature of the field. Optimization Letters aims to communicate in a timely fashion all recent developments in optimization with concise short articles (limited to a total of ten journal pages). Such concise articles will be easily accessible by readers working in any aspects of optimization and wish to be informed of recent developments.