A Reinforcement Learning Algorithm for Market Participants in FTR Auctions

Abstract

This paper presents a Q-Learning algorithm for the development of bidding strategies for market participants in FTR auctions. Each market participant is represented by an autonomous adaptive agent capable of developing its own bidding behavior based on a Q-learning algorithm. Initially, a bi- level optimization problem is formulated. At the first level, a market participant tries to maximize his expected profit under the constraint that, at the second level, an independent system operator tries to maximize the revenues from the FTR auction. It is assumed that each FTR market participant chooses his bidding strategy, for holding a FTR, based on a probabilistic estimate of the LMP differences between withdrawal and injection points. The market participant expected profit is calculated and a Q- learning algorithm is employed to find the optimal bidding strategy. A two-bus and a five-bus test system are used to illustrate the presented method.