Kuldeep R. Kurte, J. Munk, Kadir Amasyali, O. Kotevska, Borui Cui, T. Kuruganti, H. Zandi
{"title":"Electricity Pricing aware Deep Reinforcement Learning based Intelligent HVAC Control","authors":"Kuldeep R. Kurte, J. Munk, Kadir Amasyali, O. Kotevska, Borui Cui, T. Kuruganti, H. Zandi","doi":"10.1145/3427773.3427866","DOIUrl":"https://doi.org/10.1145/3427773.3427866","url":null,"abstract":"Recently, deep reinforcement learning (DRL) based intelligent control of Heating, Ventilation, and Air Conditioning (HVAC) has gained a lot of attention due to DRL's ability to optimally control HVAC for minimizing operational cost while maintaining resident's comfort. The success of such DRL-based techniques largely depends on the articulation of the problem in terms of states, actions, and reward function. Inclusion of the electricity pricing information in the problem formulation can play an important role in saving the cost of HVAC operation. However, less attention has been given in the literature on formulating well-crafted state features based on electricity pricing. In this work, we propose an approach for training the DRL model with a specific focus on feature engineering based on electricity pricing. During training, we generate random but sufficiently realistic electricity price signals so that the pre-trained DRL model is robust and adaptive to the dynamic and variable electricity prices. The validation results are encouraging and show the potential of ≈12%-15% savings in the one day cost of HVAC operation, proving the usefulness of including electricity pricing related features as state features.","PeriodicalId":351231,"journal":{"name":"Proceedings of the 1st International Workshop on Reinforcement Learning for Energy Management in Buildings & Cities","volume":"34 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115641081","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Xiangyu Zhang, Xin Jin, C. Tripp, David J. Biagioni, P. Graf, Huaiguang Jiang
{"title":"Transferable Reinforcement Learning for Smart Homes","authors":"Xiangyu Zhang, Xin Jin, C. Tripp, David J. Biagioni, P. Graf, Huaiguang Jiang","doi":"10.1145/3427773.3427865","DOIUrl":"https://doi.org/10.1145/3427773.3427865","url":null,"abstract":"To harness the great amount of untapped resources on the demand side, smart home technology plays a vital role in solving the \"last mile\" problem in smart grid. Reinforcement learning (RL), which has demonstrated an outstanding performance in solving many sequential decision-making problems, can be a great candidate to be used in smart home control. For instance, many studies have started investigating the appliance scheduling problem under dynamic pricing scheme. Based on those, this study aims at providing an affordable solution to encourage a higher smart home adoption rate. Specifically, we investigate combining transfer learning (TL) with RL to reduce the training cost of an optimal RL control policy. Given an optimal policy for a benchmark home, TL can jump-start the RL training of a policy for a new home, which has different appliances and user preferences. Simulation results show that by leveraging TL, RL training converges faster and requires much less computing time for new homes that are similar to the benchmark home. In all, this study proposes a cost-effective approach for training RL control policies for homes at scale, which ultimately reduces the controller's implementation costs, increases the adoption rate of RL controllers, and makes more homes grid-interactive.","PeriodicalId":351231,"journal":{"name":"Proceedings of the 1st International Workshop on Reinforcement Learning for Energy Management in Buildings & Cities","volume":"35 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127911302","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Sugumar Murugesan, Z. Jiang, M. J. Risbeck, J. Amores, C. Zhang, V. Ramamurti, K. H. Drees, M. LeeYoung
{"title":"Less is More: Simplified State-Action Space for Deep Reinforcement Learning based HVAC Control","authors":"Sugumar Murugesan, Z. Jiang, M. J. Risbeck, J. Amores, C. Zhang, V. Ramamurti, K. H. Drees, M. LeeYoung","doi":"10.1145/3427773.3427864","DOIUrl":"https://doi.org/10.1145/3427773.3427864","url":null,"abstract":"How do we optimize heating, ventilation and air-conditioning (HVAC) systems for energy cost and occupant comfort? How do we accomplish this in an automated fashion that adapts with time with minimal human intervention? Answers to these questions have tremendous impact on building occupant comfort, building operating costs and, importantly, environmental footprint. Understandably, this topic has received considerable attention from experts both in the industry and the academia. Among these works, deep learning, specifically deep reinforcement learning (DRL) is emerging as a data-driven control strategy without requiring an explicit dynamic model of the system. Another advantage of DRL, when successfully developed, is that it can continue to learn and adapt as the building/HVAC characteristics change with time. DRL agents, however, are challenging to train. On one hand, they may need months or years of training data (sample inefficiency), potentially inconveniencing building occupants and incurring high energy costs for a long time. On the other hand, they may converge to local optima or simply do not converge. This paper highlights one strategy to mitigate some of these challenges. We show that the choice of state and action space is as important as the choice of DRL architectures and neural network training techniques. Specifically, we formulate the HVAC control problem as a Partially Observable Markov Decision Process (POMDP), build a DRL agent using Deep Q Networks (DQN) on a building simulator, and quantify gains over a widely adopted baseline heuristic method. Subsequently, we reformulate the original problem as a restricted POMDP by severely restricting the observation (state space) and action space, and build a DRL agent for the restricted POMDP. The performance gains from this DRL agent is double that of the original agent, implying that complex state and action spaces, while information rich, can lead to complex loss functions that could not be maneuvered well by a DRL agent. Our larger message is this: 'less' (state-action space) can be 'more', in the context of DRL training.","PeriodicalId":351231,"journal":{"name":"Proceedings of the 1st International Workshop on Reinforcement Learning for Energy Management in Buildings & Cities","volume":"14 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134044743","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Gauraang Dhamankar, José R. Vázquez-Canteli, Z. Nagy
{"title":"Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms on a Building Energy Demand Coordination Task","authors":"Gauraang Dhamankar, José R. Vázquez-Canteli, Z. Nagy","doi":"10.1145/3427773.3427870","DOIUrl":"https://doi.org/10.1145/3427773.3427870","url":null,"abstract":"Periods of high demand for electricity can raise electricity prices for building users. Flattening the electricity demand curve reduces can reduce costs and increase resiliency. We formulate this task as a multi-agent reinforcement learning (MA-RL) problem, to be achieved through demand response and coordination of electricity consuming agents, i.e., buildings. Bechmarks for such MA-RL problems do not exist. Here, we contribute an empirical comparison of three classes of MA-RL algorithms: independent learners, centralized critics with decentralized execution, and value factorization learners. We evaluate these algorithms on an energy coordination task in CityLearn, an Open AI Gym environment. We found independent learners with shaped rewards to be competitive with more complex algorithms. Agents with centralized critics aim to learn a rich joint critic, which may complicate the training process and cause scalability issues. Our findings indicate value factorization learners possess the coordination benefits of centralized critics and match independent learners without individualized reward shaping.","PeriodicalId":351231,"journal":{"name":"Proceedings of the 1st International Workshop on Reinforcement Learning for Energy Management in Buildings & Cities","volume":"18 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124941189","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Continual adaptation in deep reinforcement learning-based control applied to non-stationary building environments","authors":"Avisek Naug, Marcos Quiñones-Grueiro, G. Biswas","doi":"10.1145/3427773.3427867","DOIUrl":"https://doi.org/10.1145/3427773.3427867","url":null,"abstract":"This paper develops a continual deep reinforcement relearning (RL) controller for large buildings that exhibit non-stationary behaviors. The non-stationarity in building operations caused by unexpected changes in weather patterns, occupancy, and faults, makes it imperative to develop control algorithms that adapt to the changing conditions. Given the slow time constants in building operations, we assume that the non-stationarity can be modeled as discrete transitions between stationary models of system behavior. We address the challenge of detecting transitions between stationary processes using trend analysis, and relearn control policies that accommodate the tradeoff between energy savings and comfort after such transitions occur. We demonstrate our approach on a \"smart building test-bed\" for developing data-driven HVAC controllers that are deployed in large buildings on our university campus.","PeriodicalId":351231,"journal":{"name":"Proceedings of the 1st International Workshop on Reinforcement Learning for Energy Management in Buildings & Cities","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129119143","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Joon-Yong Lee, Sen Huang, Aowabin Rahman, Amanda D. Smith, S. Katipamula
{"title":"Flexible Reinforcement Learning Framework for Building Control using EnergyPlus-Modelica Energy Models","authors":"Joon-Yong Lee, Sen Huang, Aowabin Rahman, Amanda D. Smith, S. Katipamula","doi":"10.1145/3427773.3427873","DOIUrl":"https://doi.org/10.1145/3427773.3427873","url":null,"abstract":"In recent years, reinforcement learning (RL) methods have been greatly enhanced by leveraging deep learning approaches. RL methods applied to building control have shown potential in many applications because of their ability to complement or replace conventional methods such as model-based or rule-based controls. However, RL-based building control software is likely tailored either to one target building system or to a specific RL method so that significant additional effort would be required to customize the RL-based controller for use in other building systems or with other RL approaches. Also, RL-based building controls usually depend on building energy simulations to train controllers, so emulating building dynamics (i.e., thermal dynamics and control dynamics) and capturing sub-hourly dynamic profiles are crucial to further the development of effective RL-based building control methods. To address these challenges, we present an RL-based control software employing a high-fidelity hybrid EnergyPlus-Modelica building energy model that emulates building dynamics at 1 minute resolution. This software consists of decoupled components (environment, building emulator, control agent, and RL algorithm), which allows for quick prototyping and benchmarking of standard RL algorithms in different systems; for example, a single component can be replaced without revising the software. To demonstrate this software framework, we conducted a benchmark study using an EnergyPlus-Modelica building energy model for a Chicago office building with an RL-based controller to dynamically control the chilled water temperature setpoint and the air handling unit supply air temperature setpoints.","PeriodicalId":351231,"journal":{"name":"Proceedings of the 1st International Workshop on Reinforcement Learning for Energy Management in Buildings & Cities","volume":"31 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133142383","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Demand Response through Price-setting Multi-agent Reinforcement Learning","authors":"M. Christensen, Cédric Ernewein, P. Pinson","doi":"10.1145/3427773.3427862","DOIUrl":"https://doi.org/10.1145/3427773.3427862","url":null,"abstract":"Price based demand response is a cost-effective way of obtaining flexibility needed in power systems with high penetration of intermittent renewable energy sources. Model-free deep reinforcement learning is proposed as a way to train autonomous agents for enabling buildings to participate in demand response programs as well as coordinating such programs though price setting in a multiagent setup. First, we show price responsive control of buildings with electric heat pumps using deep deterministic policy gradient. Then a coordinating agent is trained to manage a population of buildings by adjusting the price in order to keep the total load from exceeding the available capacity considering also the non-flexible base load.","PeriodicalId":351231,"journal":{"name":"Proceedings of the 1st International Workshop on Reinforcement Learning for Energy Management in Buildings & Cities","volume":"1388 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125157200","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Lucas Spangher, Akash Gokul, Manan Khattar, Joseph Palakapilly, Utkarsha Agwan, A. Tawade, C. Spanos
{"title":"Augmenting Reinforcement Learning with a Planning Model for Optimizing Energy Demand Response","authors":"Lucas Spangher, Akash Gokul, Manan Khattar, Joseph Palakapilly, Utkarsha Agwan, A. Tawade, C. Spanos","doi":"10.1145/3427773.3427863","DOIUrl":"https://doi.org/10.1145/3427773.3427863","url":null,"abstract":"While reinforcement learning (RL) on humans has shown incredible promise, it often suffers from a scarcity of data and few steps. In instances like these, a planning model of human behavior may greatly help. We present an experimental setup for the development and testing of an Soft Actor Critic (SAC) V2 RL architecture for several different neural architectures for planning models: an autoML optimized LSTM, an OLS, and a baseline model. We present the effects of including a planning model in agent learning within a simulation of the office, currently reporting a limited success with the LSTM.","PeriodicalId":351231,"journal":{"name":"Proceedings of the 1st International Workshop on Reinforcement Learning for Energy Management in Buildings & Cities","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132239355","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Anand Krishnan Prakash, Samir Touzani, M. Kiran, Shreya Agarwal, Marco Pritoni, J. Granderson
{"title":"Deep Reinforcement Learning in Buildings: Implicit Assumptions and their Impact","authors":"Anand Krishnan Prakash, Samir Touzani, M. Kiran, Shreya Agarwal, Marco Pritoni, J. Granderson","doi":"10.1145/3427773.3427868","DOIUrl":"https://doi.org/10.1145/3427773.3427868","url":null,"abstract":"As deep reinforcement learning (DRL) continues to gain interest in the smart building research community, there is a transition from simulation-based evaluations to deploying DRL control strategies in actual buildings. While the efficacy of a solution could depend on a particular implementation, there are common obstacles that developers have to overcome to deliver an effective controller. Additionally, a deployment in a physical building can invalidate some of the assumptions made during the controller development. Assumptions on the sensor placement or on the equipment behavior can quickly come undone. This paper presents some of the significant assumptions made during the development of DRL based controllers that could affect their operations in a physical building. Furthermore, a preliminary evaluation revealed that controllers developed with some of these assumptions can incur twice the expected costs when they are deployed in a building.","PeriodicalId":351231,"journal":{"name":"Proceedings of the 1st International Workshop on Reinforcement Learning for Energy Management in Buildings & Cities","volume":"104 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128195419","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
David J. Biagioni, Xiangyu Zhang, P. Graf, D. Sigler, Wesley B. Jones
{"title":"A Comparison of Model-Free and Model Predictive Control for Price Responsive Water Heaters","authors":"David J. Biagioni, Xiangyu Zhang, P. Graf, D. Sigler, Wesley B. Jones","doi":"10.1145/3427773.3427872","DOIUrl":"https://doi.org/10.1145/3427773.3427872","url":null,"abstract":"We present a careful comparison of two model-free control algorithms, Evolution Strategies (ES) and Proximal Policy Optimization (PPO), with receding horizon model predictive control (MPC) for operating simulated, price responsive water heaters. Four MPC variants are considered: a one-shot controller with perfect forecasting yielding optimal control; a limited-horizon controller with perfect forecasting; a mean forecasting-based controller; and a two-stage stochastic programming controller using historical scenarios. In all cases, the MPC model for water temperature and electricity price are exact; only water demand is uncertain. For comparison, both ES and PPO learn neural network-based policies by directly interacting with the simulated environment under the same scenarios used by MPC. All methods are then evaluated on a separate one-week continuation of the demand time series. We demonstrate that optimal control for this problem is challenging, requiring more than 8-hour lookahead for MPC with perfect forecasting to attain the minimum cost. Despite this challenge, both ES and PPO learn good general purpose policies that outperform mean forecast and two-stage stochastic MPC controllers in terms of average cost and are more than two orders of magnitude faster at computing actions. We show that ES in particular can leverage parallelism to learn a policy in under 90 seconds using 1150 CPU cores.","PeriodicalId":351231,"journal":{"name":"Proceedings of the 1st International Workshop on Reinforcement Learning for Energy Management in Buildings & Cities","volume":"10 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-11-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130949611","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}