Efficient privacy-preserving federated learning with encrypted-domain knowledge distillation

With the development of Decentralized Federated Learning (DFL), it has demonstrated significant advantages in protecting data privacy and enhancing model generalization. However, directly exchanging unprocessed model parameters in DFL not only increases communication overhead but also significantly elevates the risk of privacy leakage among participants. In this context, knowledge distillation has emerged as an effective, lightweight model sharing method to reduce transmission burdens and was once regarded as a promising solution for decentralized federated learning. Nevertheless, studies have shown that logits, the key to its knowledge transfer for classification prediction, pose a privacy risk. Logits obtained from training on private datasets can be used to reconstruct the training data. To address these concerns, this paper proposes the DKDFL framework, which integrates knowledge distillation with Fully Homomorphic Encryption (FHE) in the DFL scenario. This framework achieves efficient knowledge distillation collaboration through grouping strategies and an innovative distillation loss function tailored for the encrypted domain, ensuring both computational efficiency and logits confidentiality. Additionally, the introduction of a coordinator node further optimizes the computation process. Experimental results indicate that DKDFL performs well in terms of model accuracy, exhibiting a relatively stable training process. While ensuring privacy protection, it maintains high model accuracy. In data heterogeneity scenarios, the performance of collaborative learning based on DKDFL significantly outperforms the results of independent training by participants. Compared to another federated learning algorithm also utilizing knowledge distillation and fully homomorphic encryption, DKDFL achieves notable improvements in reducing communication costs and time overhead.