End-effect mitigation in renewable energy systems with energy storage using value function approximation of terminal energy level

Abstract

The growing integration of renewable energy sources into power systems introduces operational challenges due to their inherent uncertainty and intermittency. In particular, the end-effect remains a critical barrier to realistic long-term scheduling, where energy storage system (ESS) tends to be completely discharged near the end of the planning horizon. To address this, we propose a novel terminal energy valuation method for ESSs within a two-stage stochastic programming (2SSP) framework, integrating reinforcement learning (RL) with value function approximation. By formulating system operations as a Markov decision process, our method iteratively updates the value of the terminal energy level in ESS using the value iteration algorithm. We first employ a linear value function approximator and then enhance performance using a neural network-based approximator. Comparative experiments demonstrate that our RL-based 2SSP significantly improves long-term profits, effectively mitigates the end-effect, and outperforms existing approaches such as fixed terminal constraints, rolling horizon frameworks, and static terminal energy valuations.