DDPG-AdaptConfig: A deep reinforcement learning framework for adaptive device selection and training configuration in heterogeneity federated learning

Abstract

Federated Learning (FL) is a distributed machine learning approach that protects user privacy by collaboratively training shared models across devices without sharing their raw personal data. Despite its advantages, FL faces issues of increased convergence time and decreased accuracy due to the heterogeneity of data and systems across devices. Existing methods for solving these issues using reinforcement learning often ignore the adaptive configuration of local training hyperparameters to suit varying data characteristics and system resources. Moreover, they frequently overlook the heterogeneous information contained within local model parameters. To address these problems, we propose the DDPG-AdaptConfig framework based on Deep Deterministic Policy Gradient (DDPG) for adaptive device selection and local training hyperparameters configuration in FL to speed up convergence and ensure high model accuracy. Additionally, we develop a new actor network that integrates the transformer mechanism to extract heterogeneous information from model parameters, which assists in device selection and hyperparameters configuration. Furthermore, we introduce a clustering-based aggregation strategy to accommodate heterogeneity and prevent performance declines. Experimental results show that our DDPG-AdaptConfig achieves significant improvements over existing baselines.