Behind-the-Meter Energy Storage System Operation via Hindsight Experience Replay

Abstract

A typical use case for Behind-the-Meter Battery Energy Storage Systems (BESS) is to manage demand charges. However, the sparse nature of demand charges presents a challenge when using reinforcement learning algorithms. To address this issue, we propose a method based on Hindsight Experience Replay which overcomes the problems associated with sparse rewards in BESS operation. The main idea of the Hindsight Experience Replay is to develop an additional reward counting process from the existing trajectory which already could handle some non-sparse rewards. And the developed reward provides a direction guidance to the final operation destination, which demolish the difficulties on sparse reward training compared to conventional reinforcement learning. Our proposed method has been verified for its feasibility and effectiveness through simulations conducted in a charge station modeled after the IEEE 13-node test feeder.