MAFD: A Federated Distillation Approach with Multi-head Attention for Recommendation Tasks

Abstract

The key challenges that recommendation systems must overcome are data isolation and privacy protection issues. Federated learning can efficiently train global models using decentralized data while preserving privacy. In real-world applications, however, it is difficult to achieve high prediction accuracy due to the heterogeneity of devices, the lack of data, and the limited generalization capacity of models. In this research, we introduce a personalized federated knowledge distillation model for a recommendation system based on a multi-head attention mechanism for recommendation systems. Specifically, we first employ federated distillation to improve the performance of student models and introduce a multi-head attention mechanism to enhance user encoding information. Next, we incorporate Wasserstein distance into the objective function of combined distillation to reduce the distribution gap between teacher and student networks and also use an adaptive learning rate technique to enhance convergence. We show that the proposed approach achieves better effectiveness and robustness through benchmarks.