A Hybrid-Update Efficient Federated Learning Method Based on Multi-Teacher Knowledge Distillation in the Internet of Things

Abstract

The emergence of federated learning (FL) provides a new learning paradigm for private protection of data in the Internet of Things (IoT). However, it takes a lot of time for the server to obtain a global model with superior performance, which restricts the development of FL in the IoT. Therefore, this paper proposes a hybrid-update efficient federated learning method based on multi-teacher knowledge distillation in the Internet of Things. Firstly, considering the local training of each client, we design a data separation method of divide and conquer, which transforms data separation into a many-objective solution problem with constraints, and the unseparated data is used to train local models to speed up the training of the local model. Then, to alleviate the adverse effects of the above method, we introduce the knowledge distillation technology, and a multi-teacher model is designed for separated data. The teacher models are trained in advance, and they pass on their respective professional knowledge to the local models during the FL process. In the communication between the clients and the server, we only pass part of the model weights to further improve the overall efficiency of FL. To mitigate the impact of the above process, this paper proposes a hybrid-update federated learning strategy, which divides the update of the global model into federated aggregation update and generative weights update to improve the performance. Finally, we use the MNIST dataset, fashion-MNIST dataset, GTSRB dataset, SVHN dataset, and 20 Newsgroups dataset to simulate non-independent and identically distributed (non-IID) scenarios, and many experiments are performed to verify the effectiveness of the proposed method. Our method improves the overall efficiency of FL and further promotes the development of the IoT.