An Efficient Multi-Model Training Algorithm for Federated Learning

Abstract

How to effectively organize various heterogeneous clients for effective model training has been a critical issue in federated learning. Existing algorithms in this aspect are all for single model training and are not suitable for parallel multi-model training due to the inefficient utilization of resources at the powerful clients. In this paper, we study the issue of multi-model training in federated learning. The objective is to effectively utilize the heterogeneous resources at clients for parallel multi-model training and therefore maximize the overall training efficiency while ensuring a certain fairness among individual models. For this purpose, we introduce a logarithmic function to characterize the relationship between the model training accuracy and the number of clients involved in the training based on measurement results. We accordingly formulate the multi-model training as an optimization problem to find an assignment to maximize the overall training efficiency while ensuring a log fairness among individual models. We design a Logarithmic Fairness based Multi-model Balancing algorithm (LFMB), which iteratively replaces the already assigned models with a not-assigned model at each client for improving the training efficiency, until no such improvement can be found. Numerical results demonstrate the significantly high performance of LFMB in terms of overall training efficiency and fairness.