Reinforcement Learning Based Bidding Method with High-dimensional Bids in Electricity Markets

Abstract

Over the past decade, bidding in electricity markets has attracted widespread attention. Reinforcement learning (RL) has been widely used for electricity market bidding as a powerful artificial intelligence (AI) tool to make decisions under real-world uncertainties. However, current RL-based bidding methods mostly employ low-dimensional bids (LDBs), which significantly diverge from the $N$ price-power pairs commonly used in current electricity markets. The $N$-pair bid format is denoted as high-dimensional bid (HDB) format, which has not been fully integrated into the existing RL-based bidding methods. The loss of flexibility of current RL-based bidding methods could greatly limit the bidding profits and make it difficult to address the increasing uncertainties caused by renewable energy generation. In this paper, we propose a framework for fully utilizing HDBs in RL-based bidding methods. First, we employ a special type of neural network called the neural network supply function (NNSF) to generate HDBs in the form of $N$ price-power pairs. Second, we embed the NNSF into a Markov decision process (MDP) to make it compatible with most existing RL algorithms. Finally, the experiments on energy storage systems (ES-Ss) in the Pennsylvania-New Jersey-Maryland (PJM) real-time electricity market show that the proposed bidding method with HDBs can increase the bidding flexibility, thereby increasing the profits of state-of-the-art RL-based bidding methods.