Optimizing intelligent residential scheduling based on policy black box and adaptive clustering federated deep reinforcement learning

Abstract

In the context of user-side demand response, flexible resources in buildings such as air conditioners and electric vehicles are characterized by small individual capacities, large aggregate scales, and geographically dispersed distributions, necessitating integration by intelligence buildings (IRs). However, the optimization scheduling of IR clusters often involves detailed energy consumption data, posing privacy issues such as revealing household routines. The traditional aggregator-IRs bi-level architecture typically employs centralized or game-theoretic strategies for optimization scheduling, which struggle to balance efficiency and privacy simultaneously. To address this issue, this paper proposes a bi-level optimization scheduling strategy that balances efficiency and privacy. First, deep reinforcement learning models are established for both the aggregator and the IRs to address efficiency. Then, the trained demand response models of the IRs are encapsulated into strategy black boxes and uploaded to the aggregator’s deep reinforcement learning model. Throughout this process, the aggregator remains unaware of the user-side data, thus protecting user privacy. Additionally, considering that training IR strategy black box models is a parallel and similar process, this paper introduces the paradigm of federated learning to reduce learning costs and improve training efficiency on the IRs side. Furthermore, an adaptive clustering federated deep reinforcement learning method is proposed to address the heterogeneity of the IRs. Finally, case studies demonstrate the feasibility and effectiveness of the proposed method.