Mixed-Action Reinforcement Learning for Energy Management of Renewable-Integrated Libraries

Abstract

Energy coordination for renewable-integrated public buildings involves coupled physical dynamics, operational uncertainty, and mixed decision variables. Conventional optimization methods rely on accurate forecasts and repeated online computation, whereas standard reinforcement learning (RL) methods with continuous action spaces are not well suited to hybrid scheduling tasks involving both binary and continuous decisions. To address this issue, a proximal policy optimization (PPO)-based approach is developed for a smart library energy system with photovoltaic generation, an energy storage system (ESS), electric vehicles (EVs), and flexible tasks. The energy scheduling is formulated as a sequential decision-making problem that incorporates electricity price variation, ESS and EV charging behavior, and task scheduling requirements. A mixed binary–continuous policy is designed to unify discrete task scheduling and continuous energy control within one policy architecture, enabling direct coordination of task execution. The effectiveness of the proposed method is validated through Python-based simulations. Compared with a conventional continuous-action method, the proposed approach achieves more effective price-responsive ESS scheduling, reduces electricity purchase during high-price periods, and yields a 26.1% improvement in the overall economic index.