Swarm mutual learning

Abstract

With the rapid growth of big data, extracting meaningful knowledge from data is crucial for machine learning. The existing Swarm Learning data collaboration models face challenges such as data security, model security, high communication overhead, and model performance optimization. To address this, we propose the Swarm Mutual Learning (SML). Firstly, we introduce an Adaptive Mutual Distillation Algorithm that dynamically controls the learning intensity based on distillation weights and strength, enhancing the efficiency of knowledge extraction and transfer during mutual distillation. Secondly, we design a Global Parameter Aggregation Algorithm based on homomorphic encryption, coupled with a Dynamic Gradient Decomposition Algorithm using singular value decomposition. This allows the model to aggregate parameters in ciphertext, significantly reducing communication overhead during uploads and downloads. Finally, we validate the proposed methods on real datasets, demonstrating their effectiveness and efficiency in model updates. On the MNIST dataset and CIFAR-10 dataset, the local model accuracies reached 95.02% and 55.26%, respectively, surpassing those of the comparative models. Furthermore, while ensuring the security of the aggregation process, we significantly reduced the communication overhead for uploading and downloading.