A fast federated reinforcement learning approach with phased weight-adjustment technique

Abstract

Federated reinforcement learning (FRL) enables multiple agents to learn collaboratively without directly sharing their local data. This method addresses the data privacy concerns in the distributed systems. However, FRL faces challenges such as high communication costs, since it requires extensive interactions to achieve satisfied performance. Therefore, this paper develops a fast FRL method with a dynamic aggregation coefficient to reduce the communication load during the learning process. Diverging from traditional FRL techniques which rely on static averaging, our approach begins by setting the initial aggregation coefficient to the logarithm of the number of participating agents. This elevation can enhance the early integration of updates from distributed agents and facilitate a rapid initial learning phase. As communications progress, the aggregation coefficient linearly decreases, transitioning to an average aggregation by the end of the specified interval. This gradual reduction aligns individual learning updates more closely over time, shifting towards a unified global learning model. Furthermore, we implement a value-clipping strategy to constrain global updates within a predefined safe range, thus safeguarding against the potential overflow issues. The aggregation coefficient stabilizes after the initial aggressive integration phase to ensure the training stability. The boundedness analysis of the model aggregation confirms that, despite the high initial coefficient, the parameters of the global model remain within the manageable limits on the FRL server. This strategy is applicable to both tabular and deep learning methods. We validate the designed algorithm on navigation and control tasks, including heterogeneous environments where distinct state transitions and dynamics are designed for each agent. The experimental results demonstrate that our proposed approach achieves faster convergence across various environments.