Towards predicting client benefit and contribution in federated learning from data imbalance

Federated learning (FL) is a distributed learning paradigm that allows a cohort of clients to collaborate in jointly training a machine learning model. By design, FL assures data-privacy for clients involved, making it the perfect fit for a wide range of real-world applications requiring data privacy. Despite its great potential and conceptual guarantees, FL has been found to suffer from unbalanced data, causing the overall performance of the final model to decrease and the contribution of individual clients to the federated model to vary greatly. Assuming that imbalance does not only affect contribution but also the extent to which individual clients benefit from participating in FL, we investigate the predictive potential of data imbalance metrics on benefit and contribution. In particular, our approach comprises three phases: (1) we measure data imbalance of clients while maintaining data privacy using secure aggregation, (2) we measure how individual clients benefit from FL participation and how valuable they are for the cohort, and (3) we train classifiers to pairwisely rank clients regarding benefit and contribution. The resulting classifiers rank pairs of clients with an accuracy of 0.71 and 0.65 for benefit and contribution, respectively. Thus, our approach contributes towards providing an indication for the expected value for individual clients and the cohort prior to their participation.