A Chunked Local Aggregation Strategy in Federated Learning

Abstract

Federated Learning (FL) is a distributed machine learning technology that trains models on large-scale distributed devices while keeping training data localized and privatized. However, in settings where data is distributed in a not independent and identically distributed (non-I.I.D.) fashion, the single joint model produced by FL suffers in terms of test set accuracy and communication costs. And a multi-layer topology are widely deployed for FL in real scenarios. Therefore, we propose FedBox, a chunked local aggregation federated learning framework to improve the generalization ability and aggregation efficiency of model in non-I.I.D. data by adapting to the topology of the real network. Moreover, we study the adaptive gradient descent (AGC) to mitigate the feature shift caused by training non-I.I.D. data. In this work, we modified the aggregation strategy of FL by introducing a virtual node layer based on local stochastic gradient methods (SGD), and separate the edge node cluster by the similarity between the local update model and the global update model. We show that FedBox can effectively improve convergence speed and test accuracy, while reducing communication cost. Training results on FederatedEMNIST, Cifar10, Cifar100 and Shakespeare datasets indicate that FedBox allows model training to converge in fewer communication rounds and improves training accuracy by up to 3.1% compared with FedAVG. In addition, we make an empirical analysis of the extended range of virtual nodes.