FedELR: When federated learning meets learning with noisy labels

Abstract

Existing research on federated learning (FL) usually assumes that training labels are of high quality for each client, which is impractical in many real-world scenarios (e.g., noisy labels by crowd-sourced annotations), leading to dramatic performance degradation. In this work, we investigate noisy FL through the lens of early-time training phenomenon (ETP). Specifically, a key finding of this paper is that the early training phase varies among different local clients due to the different noisy classes in each client. In addition, we show that such an inconsistency also exists between the local and global models. As a result, local clients would always begin to memorize noisy labels before the global model reaches the optimal, which inevitably leads to the degradation of the quality of service in real-world FL applications (e.g. tumor image classification among different hospitals). Our findings provide new insights into the learning dynamics and shed light on the essence cause of this degradation in noisy FL. To address this problem, we reveal a new principle for noisy FL: it is necessary to align the early training phases across local models. To this end, we propose FedELR, a simple yet effective framework that aims to force local models to stick to their early training phase via an early learning regularization (ELR), so that the learning dynamics of local models can be kept at the same pace. Moreover, this also leverages the ETP in local clients, leading each client to take more training steps in learning a more robust local model for optimal global aggregation. Extensive experiments on various real-world datasets also validate the effectiveness of our proposed methods.