Federated proximal policy optimization with action masking: Application in collective heating systems

Abstract

This paper introduces a novel privacy-aware Federated Proximal Policy Optimization (FPPO) method combined with action masking. As a Federated Reinforcement Learning (FRL) approach, the proposed method is used for optimizing the reloading of Domestic Hot Water (DHW) storage tanks, with a focus on energy savings and DHW thermal comfort in collective heating systems. The proposed approach combines FedProx as the Federated Learning (FL) method and Proximal Policy Optimization (PPO) as the Deep Reinforcement Learning (DRL) technique to address the challenges of distributed control while ensuring data privacy. Key contributions include: (1) employing action masking to guarantee compliance with comfort level, (2) designing a global reward function to align agents actions toward collective energy savings, (3) implementing a privacy-aware design where only model parameters are shared with a global aggregator, avoiding raw data transmission, and (4) optimizing PPO’s loss function for improved performance.

PPO was benchmarked using a common FL method (FedAvg) alongside two other DRL methods, where PPO outperformed both in scalability and energy savings, especially in larger systems. Then, PPO-based FRL was refined into FPPO by integrating a proximal term with coefficient $\mu$ into the loss function to enhance the performance. Experiments were conducted with both fixed and dynamically adjusted $\mu$, with the latter demonstrating better energy savings and comfort. Results show that FPPO achieves up to 10.08% energy savings while maintaining DHW discomfort below 8.72% in systems with at least 20 dwellings. These findings highlight FPPO as a scalable, privacy-aware, and energy-efficient solution for distributed control in collective heating systems.