Learn the global prompt in the low-rank tensor space for heterogeneous federated learning

Abstract

Federated learning collaborates with multiple clients to train a global model, enhancing the model generalization while allowing the local data transmission-free and security. However, federated learning currently faces three intractable challenges: (1) The large number of model parameters result in an excessive communication burden. (2) The non-independently and identically distributed local data induces the degradation of global model. (3) The model heterogeneity renders traditional federated aggregation infeasible. To dissipate the three difficulties, we propose to learn the global prompt in the low-rank tensor space (FedGPT) for heterogeneous federated learning. Specifically, we employ the prompts rather than the model parameters as the carrier of local knowledge to achieve the information interaction between multiple clients. Since the prompts only have a very small number of variables, the communication volume is greatly reduced. To cope with the data heterogeneity, the prompts from different clients are stacked into the third-order tensors, on which the tensor singular value decomposition is performed to extract the global information. Furthermore, the proposed FedGPT possesses the ability to handle the model heterogeneity, the local models of different sizes can transfer the knowledge with the help of the prompts to improve the performance. Extensive experiments on three real-world datasets are conducted. Overall, FedGPT outperforms other state-of-the-art compared methods by up to 13.21%, and achieves less than 3% of communication volume of FedAvg, demonstrating the superiority of the proposed FedGPT.