Trade-Off Between Renewable Energy Utilizing and Communication Quality for Base Stations: A Heterogeneous Multi-Agent Safe RL Method

Abstract

The ultra-dense deployment of base stations (BSs) results in significant energy costs, while the increasing use of fluctuating renewable energy sources (RESs) threatens the safe operation of electric network (EN). These issues can be addressed by coordinating BSs’ active/sleep states with RES generation. However, the coordinated decision-making is challenging due to the conflicting goals of maximizing RES utilization, guaranteeing communication quality of service (QoS). In this paper, we design an electric-cellular collaborative network (ECCN) and formulate a joint optimization problem to minimize electric supply and QoS degradation costs, subjecting to EN’s safety constraints. Considering the uncertainty of RES generation and BS traffic, we propose a heterogeneous multi-agent safe reinforcement learning (HMAS-RL) algorithm to solve the problem. HMAS-RL inherits from the centralized critic and decentralized actor framework, where two heterogeneous agents learn a cooperative policy for joint decision-making through two actor networks design that share observations and a team reward. A cost critic network is designed to handle safety constraints, eliminating the need for manual penalty term design and tuning. We validate the proposed method using an IEEE 33-bus electric distribution test feeder. Results demonstrate HMAS-RL achieves superior performance in RES utilization, communication QoS, and EN safety constraints maintenance.