Sketch-Based Adaptive Communication Optimization in Federated Learning

Abstract

In recent years, cross-device federated learning (FL), particularly in the context of Internet of Things (IoT) applications, has demonstrated its remarkable potential. Despite significant efforts, empirical evidence suggests that FL algorithms have yet to gain widespread practical adoption. The primary obstacle stems from the inherent bandwidth overhead associated with gradient exchanges between clients and the server, resulting in substantial delays, especially within communication networks. To deal with the problem, various solutions are proposed with the hope of finding a better balance between efficiency and accuracy. Following this goal, we focus on investigating how to design a lightweight FL algorithm that requires less communication cost while maintaining comparable accuracy. Specifically, we propose a Sketch-based FL algorithm that combines the incremental singular value decomposition (ISVD) method in a way that does not negatively affect accuracy much in the training process. Moreover, we also provide adaptive gradient error accumulation and error compensation mechanisms to mitigate accumulated gradient errors caused by sketch compression and improve the model accuracy. Our extensive experimentation with various datasets demonstrates the efficacy of our proposed approach. Specifically, our scheme achieves nearly a 93% reduction in communication cost during the training of multi-layer perceptron models (MLP) using the MNIST dataset.