压缩卷积神经网络中张量分解的自动选择——以vgg型网络为例

Chia-Chun Liang, Che-Rung Lee
{"title":"压缩卷积神经网络中张量分解的自动选择——以vgg型网络为例","authors":"Chia-Chun Liang, Che-Rung Lee","doi":"10.1109/IPDPSW52791.2021.00115","DOIUrl":null,"url":null,"abstract":"Tensor decomposition is one of the model reduction techniques for compressing deep neural networks. Existing methods use either Tucker decomposition (TD) or Canonical Polyadic decomposition (CPD) for model compression, but none of them tried to combine those two methods, owing to the complexity of choosing a proper decomposition method for each layer. In this paper, we adopted the automatic tuning technique to design an algorithm that can mix both tensor decomposition methods, called Mixed Tensor Decomposition (MTD). The goal is to achieve better compression ratio while keeping similar accuracy as the original models. We used VGG type networks for the case study since they are relatively heavy and computationally expensive. We first studied the relation of model accuracy and compression ratio for Tucker and CPD applying to convolution neural networks (CNN). Based on the studied results, we designed a strategy to select the most suitable decomposition method for each layer, and further fine-tunes the models to recover the accuracy. We have conducted experiments using VGG11 and VGG16 with CIFAR10 dataset, and compared MTD with other tensor decomposition algorithms. The results show that MTD can achieve compression ratio 32 × and 37 × for VGG11 and VGG16 respectively with less than 1% accuracy drops, which is much better than the state-of-the-art tensor decomposition algorithms for model compression.","PeriodicalId":170832,"journal":{"name":"2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)","volume":"22 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"1","resultStr":"{\"title\":\"Automatic Selection of Tensor Decomposition for Compressing Convolutional Neural Networks A Case Study on VGG-type Networks\",\"authors\":\"Chia-Chun Liang, Che-Rung Lee\",\"doi\":\"10.1109/IPDPSW52791.2021.00115\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Tensor decomposition is one of the model reduction techniques for compressing deep neural networks. Existing methods use either Tucker decomposition (TD) or Canonical Polyadic decomposition (CPD) for model compression, but none of them tried to combine those two methods, owing to the complexity of choosing a proper decomposition method for each layer. In this paper, we adopted the automatic tuning technique to design an algorithm that can mix both tensor decomposition methods, called Mixed Tensor Decomposition (MTD). The goal is to achieve better compression ratio while keeping similar accuracy as the original models. We used VGG type networks for the case study since they are relatively heavy and computationally expensive. We first studied the relation of model accuracy and compression ratio for Tucker and CPD applying to convolution neural networks (CNN). Based on the studied results, we designed a strategy to select the most suitable decomposition method for each layer, and further fine-tunes the models to recover the accuracy. We have conducted experiments using VGG11 and VGG16 with CIFAR10 dataset, and compared MTD with other tensor decomposition algorithms. The results show that MTD can achieve compression ratio 32 × and 37 × for VGG11 and VGG16 respectively with less than 1% accuracy drops, which is much better than the state-of-the-art tensor decomposition algorithms for model compression.\",\"PeriodicalId\":170832,\"journal\":{\"name\":\"2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)\",\"volume\":\"22 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2021-06-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"1\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1109/IPDPSW52791.2021.00115\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/IPDPSW52791.2021.00115","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 1

摘要

张量分解是一种用于压缩深度神经网络的模型约简技术。现有的模型压缩方法要么使用Tucker分解(TD),要么使用Canonical Polyadic分解(CPD),但由于对每一层选择合适的分解方法的复杂性,没有一种方法试图将这两种方法结合起来。本文采用自动调优技术设计了一种混合两种张量分解方法的算法,称为混合张量分解(MTD)。目标是获得更好的压缩比,同时保持与原始模型相似的精度。我们在案例研究中使用了VGG类型的网络,因为它们相对较重且计算成本较高。我们首先研究了应用于卷积神经网络(CNN)的Tucker和CPD模型精度与压缩比的关系。在此基础上,我们设计了一种策略,为每一层选择最合适的分解方法,并进一步微调模型以恢复精度。我们在CIFAR10数据集上使用VGG11和VGG16进行了实验,并将MTD与其他张量分解算法进行了比较。结果表明,MTD对VGG11和VGG16的压缩比分别为32 ×和37 ×,精度下降幅度小于1%,大大优于目前最先进的张量分解模型压缩算法。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
Automatic Selection of Tensor Decomposition for Compressing Convolutional Neural Networks A Case Study on VGG-type Networks
Tensor decomposition is one of the model reduction techniques for compressing deep neural networks. Existing methods use either Tucker decomposition (TD) or Canonical Polyadic decomposition (CPD) for model compression, but none of them tried to combine those two methods, owing to the complexity of choosing a proper decomposition method for each layer. In this paper, we adopted the automatic tuning technique to design an algorithm that can mix both tensor decomposition methods, called Mixed Tensor Decomposition (MTD). The goal is to achieve better compression ratio while keeping similar accuracy as the original models. We used VGG type networks for the case study since they are relatively heavy and computationally expensive. We first studied the relation of model accuracy and compression ratio for Tucker and CPD applying to convolution neural networks (CNN). Based on the studied results, we designed a strategy to select the most suitable decomposition method for each layer, and further fine-tunes the models to recover the accuracy. We have conducted experiments using VGG11 and VGG16 with CIFAR10 dataset, and compared MTD with other tensor decomposition algorithms. The results show that MTD can achieve compression ratio 32 × and 37 × for VGG11 and VGG16 respectively with less than 1% accuracy drops, which is much better than the state-of-the-art tensor decomposition algorithms for model compression.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信