Decentralized Learning Based on Gradient Coding With Compressed Communication

Abstract

This paper considers the problem of decentralized learning (DEL) with stragglers under the communication bottleneck. In the literature, various gradient coding techniques have been proposed for distributed learning with stragglers by letting the devices transmit encoded gradients based on redundant training data. However, those techniques can not be directly applied to fully decentralized scenarios as considered in this paper due to the lack of a global model in DEL. To overcome this shortcoming, we first propose a new go ssip-based DEL method with gradient co ding (GOCO). In GOCO, to mitigate the negative impact of stragglers, the devices update the parameter vectors with encoded gradients based on stochastic gradient coding before averaging in a gossip-based manner. To further reduce the communication overhead associated with GOCO, we propose an enhanced version of GOCO, namely GOCO with compressed communication (2-GOCO), where the devices transmit compressed messages instead of the raw parameter vectors. The convergence of the proposed methods is analyzed for strongly convex loss functions. Simulation results demonstrate that the proposed methods outperform the baseline methods, which attain better learning performance under the same communication overhead.