Federated Reinforcement Learning for decentralized peer-to-peer energy trading

Abstract

The rapid development of distributed energy resources has led to an increasing number of prosumers enhancing their energy utilization, thereby raising the demands on energy management technologies. As a result, the development of future smart grids is becoming increasingly important, with a particular emphasis on integrating demand-side flexibility into electricity market. To facilitate distributed interaction among prosumers, the double-side auction market enables peer-to-peer (P2P) energy trading, maximizing the social welfare within the dynamic local electricity market. In this setup, prosumers can set their own bidding prices and optimize their operations and trading strategies. However, trading in double-side auction market faces limitations due to the complexity of the market clearing algorithm and the difficulty of predicting other participants’ bidding behaviors. To address these challenges, this paper models the P2P energy trading problem in the double-side auction market as a multi-agent reinforcement learning (MARL) task. The concept of federated learning is introduced to enhance scalability among market participants while protecting the private information of individual prosumers. Additionally, the parameter-sharing framework is proposed to accelerate the learning process. To further improve the stability of MARL training, the global information of P2P energy trading price is integrated into the critic network. The proposed federated MARL algorithm is evaluated using a real-world open-source dataset from an European residential community of 250 households with a 15-minute resolution. The evaluation assesses both the training performance of the algorithm as well as the economic and operational benefits of the P2P energy trading market compared to a traditional electricity retail market.