Optimization skill: How good is your energy management strategy really?

Abstract

This paper critically examines current evaluation practices for energy management strategies within buildings and microgrids, highlighting challenges in understanding the performance of individual models and in enabling fair comparisons across studies. To address these issues, two solutions are proposed. First, there should be a standardization of baseline models tailored to specific applications, ensuring fair evaluation and comparison. For residential battery systems, Self-Consumption Maximization (SCM) is identified as a strong candidate due to its simplicity and consistently good performance, within 8.4% of optimality, but suitable and consensual baselines must be found for other problems. Second, we propose a shift in model evaluation that positions each approach with respect to its baseline and optimal scenarios, enabling fairer inter-comparability between different works. To accomplish this, a novel metric, Optimization Skill (OS), is introduced, inspired by the Forecast Skill, which is standard in forecasting literature. OS provides a clearer perspective on model performance: for instance, while a traditional evaluation might report that a given strategy improves upon the baseline by 5.1%, the proposed framework first recognizes that the maximum possible improvement is only 5.6%, and then concludes that the model captures 90.4% of the available potential. A relevant caveat is that the applicability of OS is limited to cases where it is computationally feasible to determine the optimal solution via an optimization algorithm with perfect foresight.