FedFAA: knowledge filtering for adaptive model aggregation in federated learning

Abstract

In federated learning, performing knowledge distillation on unlabeled proxy data is an effective way to aggregate local models into a global model. Most distillation-based methods assume that all knowledge from local models is contributory, and thereby indiscriminately transfer it to the global model. However, this assumption does not hold in data heterogeneity scenarios. Incorporating noisy knowledge during the distillation can negatively impact the performance of the global model. While filtering the knowledge be transferred is an intuitive solution, performing such filtering in federated learning is challenging due to the lack of available proxy-sample labels for knowledge validation. To address this issue, we propose a knowledge filtering approach for adaptive local model aggregation (FedFAA), which filters the knowledge before distillation based on its relevance. Specifically, we design a scoring method that exploits the representation space of a model to measure the relevance between the model knowledge and each proxy sample, without relying on validation labels. With these relevance scores, we further introduce an adaptive teacher model selection scheme that maintains an appropriate distribution of knowledge-providing teacher models across proxy samples, balancing the precision and diversity of the transferred knowledge after filtering. Theoretical analysis and extensive experiments demonstrate the effectiveness of our approach and its superior performance over six state-of-the-art methods.