{"title":"DC-S3GD: Delay-Compensated Stale-Synchronous SGD for Large-Scale Decentralized Neural Network Training","authors":"Alessandro Rigazzi","doi":"10.1109/DLS49591.2019.00013","DOIUrl":null,"url":null,"abstract":"Data parallelism has become the de facto standard for training Deep Neural Network on mul- tiple processing units. In this work we propose DC- S3GD, a decentralized (without Parameter Server) stale-synchronous version of the Delay-Compensated Asynchronous Stochastic Gradient Descent (DC- ASGD) algorithm. In our approach, we allow for the overlap of computation and communication, by averaging in parameter space and compensating the inherent error with a first-order correction of the locally computed gradients. We prove the effectiveness of our approach by training Convolutional Neural Network with large batches and achieving state-of- the-art results.","PeriodicalId":363604,"journal":{"name":"2019 IEEE/ACM Third Workshop on Deep Learning on Supercomputers (DLS)","volume":"28 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2019-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"3","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2019 IEEE/ACM Third Workshop on Deep Learning on Supercomputers (DLS)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/DLS49591.2019.00013","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 3
Abstract
Data parallelism has become the de facto standard for training Deep Neural Network on mul- tiple processing units. In this work we propose DC- S3GD, a decentralized (without Parameter Server) stale-synchronous version of the Delay-Compensated Asynchronous Stochastic Gradient Descent (DC- ASGD) algorithm. In our approach, we allow for the overlap of computation and communication, by averaging in parameter space and compensating the inherent error with a first-order correction of the locally computed gradients. We prove the effectiveness of our approach by training Convolutional Neural Network with large batches and achieving state-of- the-art results.