Deep Reinforcement Learning for Online Scheduling of Photovoltaic Systems with Battery Energy Storage Systems

Abstract

A new online scheduling algorithm is proposed for photovoltaic (PV) systems with battery-assisted energy storage systems (BESS). The stochastic nature of renewable energy sources necessitates the employment of BESS to balance energy supplies and demands under uncertain weather conditions. The proposed online scheduling algorithm aims at minimizing the overall energy cost by performing actions such as load shifting and peak shaving through carefully scheduled BESS charging/discharging activities. The scheduling algorithm is developed by using deep deterministic policy gradient (DDPG), a deep reinforcement learning (DRL) algorithm that can deal with continuous state and action spaces. One of the main contributions of this work is a new DDPG reward function, which is designed based on the unique behaviors of energy systems. The new reward function can guide the scheduler to learn the appropriate behaviors of load shifting and peak shaving through a balanced process of exploration and exploitation. The new scheduling algorithm is tested through case studies using real world data, and the results indicate that it outperforms existing algorithms such as Deep Q-learning. The online algorithm can efficiently learn the behaviors of optimum non-casual off-line algorithms.