Data-Free Knowledge Filtering and Distillation in Federated Learning

Abstract

In federated learning (FL), multiple parties collaborate to train a global model by aggregating their local models while keeping private training sets isolated. One problem hindering effective model aggregation is data heterogeneity. Federated ensemble distillation tackles this problem by using fused local-model knowledge to train the global model rather than directly averaging model parameters. However, most existing methods fuse all knowledge indiscriminately, which makes the global model inherit some data-heterogeneity-caused flaws from local models. While knowledge filtering is a potential coping method, its implementation in FL is challenging due to the lack of public data for knowledge validation. To address this issue, we propose a novel data-free approach (FedKFD) that synthesizes credible labeled data to support knowledge filtering and distillation. Specifically, we construct a prediction capability description to characterize the samples where a local model makes correct predictions. FedKFD explores the intersection of local-model-input space and prediction capability descriptions with a conditional generator to synthesize consensus-labeled proxy data. With these labeled data, we filter for relevant local-model knowledge and further train a robust global model through distillation. The theoretical analysis and extensive experiments demonstrate that our approach achieves improved generalization, superior performance, and compatibility with other FL efforts.