A Stochastic Asynchronous Gradient Descent Algorithm with Delay Compensation Mechanism

Abstract

A large amount of idle computing power exists in mobile devices, which can be deployed with large-scale machine learning applications. One of the key problems is how to reduce the communication overhead between different nodes. In recent years, gradient sparsity is introduced to reduce the communication overhead. However, in the federated learning scenario, the traditional synchronous gradient optimization algorithm can not adapt to the complex network environment and high communication costs. In this paper, we propose a stochastic gradient descent algorithm with delay compensation mechanism (FedDgd) for asynchronous distributed training and further optimize it for federated asynchronous training. It is proved theoretically that FedDgd can converge at the same rate as ASGD for non-convex neural networks. Moreover, FedDgd way converge quickly and tolerates staleness in various app applications as well.