Stabilizing and improving federated learning with highly non-iid data and client dropout

Abstract

The label distribution skew has been shown to be a significant obstacle that limits the model performance in federated learning (FL). This challenge could be more serious when the participating clients are in unstable network circumstances and drop out frequently. Previous works have demonstrated that the classifier head is particularly sensitive to the label skew. Therefore, maintaining a balanced classifier head is of significant importance for building a good and unbiased global model. To this end, we propose a simple yet effective framework by introducing a calibrated softmax function with smoothed prior for computing the cross-entropy loss, and a prototype-based feature augmentation scheme to re-balance the local training, which provide a new perspective on tackling the label distribution skew in FL and are lightweight for edge devices and can facilitate the global model aggregation. With extensive experiments on two benchmark classification tasks of Fashion-MNIST and CIFAR-10, our numerical results demonstrate that our proposed method can consistently outperform the baselines, 2 8% of accuracy over FedAvg in the presence of severe label skew and client dropout.