A Code-Based Distributed Gradient Descent Method

Abstract

Distributed gradient descent is an optimization algorithm that is used to solve a minimization problem distributed over a network through minimizing local functions that sum up to form the overall objective function. These local functions fi contribute to local gradients adding up incrementally to form the overall gradient. Recently, the gradient coding paradigm was introduced for networks with a centralized fusion center to resolve the problem of straggler nodes. Through introducing some kind of redundancy on each node, such coding schemes are utilized to form new coded local functions gi from the original local functions fi. In this work, we consider a distributed network with a defined network topology and no fusion center. At each node, linear combinations of the local coded gradients $\nabla\overline{g}_{i}$ can be constructed to form the overall gradient. Our iterative method, referred to as Code-Based Distributed Gradient Descent (CDGD), updates each node's local estimate by applying an adequate weighing scheme. This scheme adapts the coded local gradient descent step along with local estimates from neighboring nodes. We provide the convergence analysis for CDGD and we analytically show that we enhance the convergence rate by a scaling factor over conventional incremental methods without any predefined tuning. Furthermore, we demonstrate through numerical results significant performance and enhancements for convergence rates.