Accelerating decentralized federated learning via momentum GD with heterogeneous delays

Abstract

Federated learning (FL) with synchronous model aggregation suffers from the straggler issue because of heterogeneous transmission and computation delays among different agents. In mobile wireless networks, this issue is exacerbated by time-varying network topology due to agent mobility. Although asynchronous FL can alleviate straggler issues, it still faces critical challenges in terms of algorithm design and convergence analysis because of dynamic information update delay (IU-Delay) and dynamic network topology. To tackle these challenges, we propose a decentralized FL framework based on gradient descent with momentum, named decentralized momentum federated learning (DMFL). We prove that DMFL is globally convergent on convex loss functions under the bounded time-varying IU-Delay, as long as the network topology is uniformly jointly strongly connected. Moreover, DMFL does not impose any restrictions on the data distribution over agents. Extensive experiments are conducted to verify DMFL’s performance superiority over the benchmarks and to reveal the effects of diverse parameters on the performance of the proposed algorithm.