Parameterized data-free knowledge distillation for heterogeneous federated learning

Abstract

Knowledge distillation has emerged as a widely adopted and effective method for addressing two challenges of heterogeneous federated learning: Data heterogeneity causes client drift, which makes model convergence slow and model accuracy decrease, and personalized requirements of heterogeneous clients are ignored, which cannot be satisfied by a single global model. However, most existing knowledge distillation-based federated learning schemes are constrained by two fundamental limitations: They rely on public datasets for knowledge distillation, forming an impractical assumption for real-world scenarios, and the model personalization process employs a unified redundant teacher model, which conflicts with the diverse data distribution characteristics among heterogeneous clients. Therefore, we propose a parameterized data-free knowledge distillation, addressing the impractical dependency on public datasets and the static single knowledge transfer bottleneck through global view knowledge extraction without public datasets and an adaptive personalized teacher model. Specifically, the server learns a conditional distribution to extract knowledge about the global view of ground-truth data distributions and then uses the acquired knowledge as an inductive bias to enhance the generalization performance of local models. Additionally, the server calculates the knowledge contribution of each local model based on the similarity of the average data representation and aggregates a personalized teacher model that contains more positive transfer knowledge for each client. Experimental validation shows that the proposed scheme improves local test accuracy by up to 69.55%, 47.56%, and 18.76% on the Mnist, EMnist, and CelebA datasets, respectively, while reducing communication rounds across varying degrees of data heterogeneity compared to state-of-the-art schemes.