Parameter-oriented contrastive schema and multi-level knowledge distillation for heterogeneous federated learning

Abstract

Federated learning aims to unite multiple data owners to collaboratively train a machine learning model without leaking the private data. However, the non-independent identically distributed (Non-IID) data differentiates the optimization directions of different clients, thus seriously impairing the performance of global model. Most efforts handling the data heterogeneity focus on the server or client side, adopting certain strategies to mitigate the differences of local models. These single-side solutions are limited in addressing the negative impact of heterogeneous data. In this paper, we attempt to overcome the problem of heterogenous federated learning simultaneously from dual sides. Specifically, to prevent the catastrophical forgetting of global information, we devise a parameter-oriented contrastive schema for correcting the optimization directions of local models on the client-side. Furthermore, considering that the only average of very diverse network parameters might damage the structural information, a multi-level knowledge distillation manner to repair the corrupt information of the global model is performed on the server-side. A multitude of experiments on four benchmark datasets demonstrate that the proposed method outperforms the state-of-the-art federated learning approaches on the Non-IID data.