FedCOLA: Federated learning with heterogeneous feature concatenation and local acceleration for non-IID data

Abstract

Federated Learning (FL) is an emerging training framework for machine learning to protect data privacy without accessing the original data from each client. However, the participating clients have different computing resources in FL. Clients with insufficient resources may not cooperate in the training due to hardware limitations. The restricted computing speeds may also slow down the overall computing time. In addition, the Non-IID problem happens when data distributions of the clients are varied, which results in lower performance for training. To overcome these problems, this paper proposes a FedCOLA approach to adapt various data distributions among heterogeneous clients. By introducing the feature concatenation and local update mechanism, FedCOLA supports different clients to train the model with different layers. Both communication load and time delay during collaborative training can be reduced. Combined with the adaptive auxiliary model and the personalized model, FedCOLA further improves the testing accuracy under various Non-IID data distributions. To evaluate the performance, this paper considers the effects and analysis of different Non-IID data distributions on distinct methods. The empirical results show that FedCOLA improves the accuracy by 5%, reduces 57% rounds to achieve the same accuracy, and reduces the communication load by 77% in the extremely imbalanced data distribution. Compared with the state-of-the-art methods in a real deployment of heterogeneous clients, FedCOLA reduces the time consumption by 70% to achieve the same accuracy and by 30% to complete 200 training rounds. In conclusion, the proposed FedCOLA not only accommodates various Non-IID data distributions but also supports the heterogeneous clients to train the model of different layers with a significant reduction of the time delay and communication load.