开发数据隔离分布式深度学习的规范化方案

IF 1.7 Q3 COMPUTER SCIENCE, INFORMATION SYSTEMS
Yujue Zhou, Ligang He, Shuang-Hua Yang
{"title":"开发数据隔离分布式深度学习的规范化方案","authors":"Yujue Zhou,&nbsp;Ligang He,&nbsp;Shuang-Hua Yang","doi":"10.1049/cps2.12004","DOIUrl":null,"url":null,"abstract":"<p>Distributed deep learning is an important and indispensable direction in the field of deep learning research. Earlier research has proposed many algorithms or techniques on accelerating distributed neural network training. This study discusses a new distributed training scenario, namely data isolated distributed deep learning. Specifically, each node has its own local data and cannot be shared for some reasons. However, in order to ensure the generalization of the model, the goal is to train a global model that required learning all the data, not just based on data from a local node. At this time, distributed training with data isolation is needed. An obvious challenge for distributed deep learning in this scenario is that the distribution of training data used by each node could be highly imbalanced because of data isolation. This brings difficulty to the normalization process in neural network training, because the traditional batch normalization (BN) method will fail under this kind of data imbalanced scenario. At this time, distributed training with data isolation is needed. Aiming at such data isolation scenarios, this study proposes a comprehensive data isolation deep learning scheme. Specifically, synchronous stochastic gradient descent algorithm is used for data exchange during training, and provides several normalization approaches to the problem of BN failure caused by data imbalance. Experimental results show the efficiency and accuracy of the proposed data isolated distributed deep learning scheme.</p>","PeriodicalId":36881,"journal":{"name":"IET Cyber-Physical Systems: Theory and Applications","volume":"6 3","pages":"105-115"},"PeriodicalIF":1.7000,"publicationDate":"2021-04-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ietresearch.onlinelibrary.wiley.com/doi/epdf/10.1049/cps2.12004","citationCount":"0","resultStr":"{\"title\":\"Developing normalization schemes for data isolated distributed deep learning\",\"authors\":\"Yujue Zhou,&nbsp;Ligang He,&nbsp;Shuang-Hua Yang\",\"doi\":\"10.1049/cps2.12004\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p>Distributed deep learning is an important and indispensable direction in the field of deep learning research. Earlier research has proposed many algorithms or techniques on accelerating distributed neural network training. This study discusses a new distributed training scenario, namely data isolated distributed deep learning. Specifically, each node has its own local data and cannot be shared for some reasons. However, in order to ensure the generalization of the model, the goal is to train a global model that required learning all the data, not just based on data from a local node. At this time, distributed training with data isolation is needed. An obvious challenge for distributed deep learning in this scenario is that the distribution of training data used by each node could be highly imbalanced because of data isolation. This brings difficulty to the normalization process in neural network training, because the traditional batch normalization (BN) method will fail under this kind of data imbalanced scenario. At this time, distributed training with data isolation is needed. Aiming at such data isolation scenarios, this study proposes a comprehensive data isolation deep learning scheme. Specifically, synchronous stochastic gradient descent algorithm is used for data exchange during training, and provides several normalization approaches to the problem of BN failure caused by data imbalance. Experimental results show the efficiency and accuracy of the proposed data isolated distributed deep learning scheme.</p>\",\"PeriodicalId\":36881,\"journal\":{\"name\":\"IET Cyber-Physical Systems: Theory and Applications\",\"volume\":\"6 3\",\"pages\":\"105-115\"},\"PeriodicalIF\":1.7000,\"publicationDate\":\"2021-04-02\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://ietresearch.onlinelibrary.wiley.com/doi/epdf/10.1049/cps2.12004\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"IET Cyber-Physical Systems: Theory and Applications\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://onlinelibrary.wiley.com/doi/10.1049/cps2.12004\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q3\",\"JCRName\":\"COMPUTER SCIENCE, INFORMATION SYSTEMS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"IET Cyber-Physical Systems: Theory and Applications","FirstCategoryId":"1085","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1049/cps2.12004","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"COMPUTER SCIENCE, INFORMATION SYSTEMS","Score":null,"Total":0}
引用次数: 0

摘要

分布式深度学习是深度学习研究中一个重要而不可或缺的方向。早期的研究提出了许多加速分布式神经网络训练的算法或技术。本研究讨论了一种新的分布式训练场景,即数据隔离分布式深度学习。具体来说,每个节点都有自己的本地数据,由于某些原因无法共享。然而,为了保证模型的泛化,我们的目标是训练一个需要学习所有数据的全局模型,而不仅仅是基于局部节点的数据。此时,需要使用数据隔离的分布式训练。在这种情况下,分布式深度学习面临的一个明显挑战是,由于数据隔离,每个节点使用的训练数据的分布可能高度不平衡。这给神经网络训练中的归一化过程带来了困难,因为传统的批归一化(BN)方法在这种数据不平衡的情况下会失效。此时,需要使用数据隔离的分布式训练。针对此类数据隔离场景,本研究提出了一种综合的数据隔离深度学习方案。具体而言,采用同步随机梯度下降算法进行训练过程中的数据交换,并针对数据不平衡导致的BN故障问题提供了几种归一化方法。实验结果表明了所提出的数据隔离分布式深度学习方案的有效性和准确性。
本文章由计算机程序翻译,如有差异,请以英文原文为准。

Developing normalization schemes for data isolated distributed deep learning

Developing normalization schemes for data isolated distributed deep learning

Distributed deep learning is an important and indispensable direction in the field of deep learning research. Earlier research has proposed many algorithms or techniques on accelerating distributed neural network training. This study discusses a new distributed training scenario, namely data isolated distributed deep learning. Specifically, each node has its own local data and cannot be shared for some reasons. However, in order to ensure the generalization of the model, the goal is to train a global model that required learning all the data, not just based on data from a local node. At this time, distributed training with data isolation is needed. An obvious challenge for distributed deep learning in this scenario is that the distribution of training data used by each node could be highly imbalanced because of data isolation. This brings difficulty to the normalization process in neural network training, because the traditional batch normalization (BN) method will fail under this kind of data imbalanced scenario. At this time, distributed training with data isolation is needed. Aiming at such data isolation scenarios, this study proposes a comprehensive data isolation deep learning scheme. Specifically, synchronous stochastic gradient descent algorithm is used for data exchange during training, and provides several normalization approaches to the problem of BN failure caused by data imbalance. Experimental results show the efficiency and accuracy of the proposed data isolated distributed deep learning scheme.

求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
IET Cyber-Physical Systems: Theory and Applications
IET Cyber-Physical Systems: Theory and Applications Computer Science-Computer Networks and Communications
CiteScore
5.40
自引率
6.70%
发文量
17
审稿时长
19 weeks
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信