Asynchronous Multi-Agent Reinforcement Learning-Based Framework for Bi-Level Noncooperative Game-Theoretic Demand Response

Demand response (DR) is proposed to address the uncertainty of distributed energy resources (DER) and energy storage systems in distribution networks. However, current research neglects the temporal relationships and the noncooperative relationship of demand response participants. A novel bi-level non-cooperative Stackelberg-Nash dynamic game-theoretic framework is proposed in this paper, specifically designed to address the issue of overlooking consumers’ interests and temporal relationships of participant actions in traditional demand response frameworks. Within this innovative framework, an asynchronous multi-agent reinforcement learning algorithm is proposed and named as Stackelberg-Nash multi-agent proximal policy optimization algorithm (SN-MAPPO). SN-MAPPO allows utility company (UC) and consumers to maximize their utilities within the proposed framework, ultimately leading to the convergence of UC and consumers’ strategies to Stackelberg-Nash equilibrium. To evaluate the effectiveness of the proposed framework, simulations are conducted using authentic data, involving one UC and eight power consumers. Simulation results confirm that the proposed demand response mechanism, while safeguarding consumers’ benefits, fosters the integration of distributed energy resource and effectively mitigates load fluctuations.