CGKDFL: A Federated Learning Approach Based on Client Clustering and Generator-Based Knowledge Distillation for Heterogeneous Data

Abstract

In practical, real-world complex networks, data distribution is frequently decentralized and Non-Independently Identically Distributed (Non-IID). This heterogeneous data presents a significant challenge for federated learning. Such problems include the generation of biased global models, the lack of sufficient personalization capability of local models, and the difficulty in absorbing global knowledge. We propose a Federated Learning Approach Based on Client Clustering and Generator-based Knowledge Distillation(CGKDFL) for heterogeneous data. Firstly, to reduce the global model bias, we propose a clustering federated learning approach that only requires each client to transmit some of the parameters of the selected layer, thus reducing the number of parameters. Subsequently, to circumvent the absence of global knowledge resulting from clustering, a generator designed to improve privacy features and increase diversity is developed on the server side. This generator produces feature representation data that aligns with the specific tasks of the client by utilizing the labeling information provided by the client. This is achieved without the need for any external dataset. The generator then transfers its global knowledge to the local model. The client can then utilize this information for knowledge distillation. Finally, extensive experiments were conducted on three heterogeneous datasets. The results demonstrate that CGKDFL outperforms the baseline method by a minimum of $7.24\%$, $6.73\%$, and $3.13\%$ regarding accuracy on the three heterogeneous datasets. Additionally, it outperforms the compared methods regarding convergence speed in all cases.