Accelerating Federated Codistillation via Adaptive Computation Amount at Network Edge

Abstract

The advent of Federated Learning (FL) empowers IoT devices to collectively train a shared model without local data exposure. In order to address the issue of Non-IID that causes model performance degradation, the recently proposed federated codistillation framework has shown great potential. However, due to the system heterogeneity of devices, the federated codistillation framework still faces a synchronization barrier issue, resulting in a non-negligible waiting time with a fixed computation amount (epoch or batch size) assigned. In this paper, we propose Adaptive Computation Amount Allocation (ACAA) to accelerate federated codistillation. Specifically, we leverage a criterion, solution inexactness, to quantify the computation amount. We dynamically adjust the solution inexactness of devices based on their computing power and bandwidth to enable them nearly simultaneous completion of training, reducing synchronization waiting time without sacrificing the training performance. The minimum required computation amount is determined by the coefficient of the distillation term and the gradient dissimilarity bound of Non-IID. We theoretically analyze the convergence of ACAA. Extensive experiments show that, compared to benchmark algorithms, ACAA can accelerate training by up to 5×.