A Federated Deep Reinforcement Learning-based Low-power Caching Strategy for Cloud-edge Collaboration

Abstract

In the era of ubiquitous network devices, an exponential increase in content requests from user equipment (UE) calls for optimized caching strategies within a cloud-edge integration. This approach is critical to handling large numbers of requests. To enhance caching efficiency, federated deep reinforcement learning (FDRL) is widely used to adjust caching policies. Nonetheless, for improved adaptability in dynamic scenarios, FDRL generally demands extended and online deep training, incurring a notable energy overhead when contrasted with rule-based approaches. With the aim of achieving a harmony between caching efficiency and training energy expenditure, we integrate a content request latency model, a deep reinforcement learning model based on markov decision processes (MDP), and a two-stage training energy consumption model. Together, these components define a new average delay and training energy gain (ADTEG) challenge. To address this challenge, we put forth a innovative dynamic federated optimization strategy. This approach refines the pre-training phase through the use of cluster-based strategies and parameter transfer methodologies. The online training phase is improved through a dynamic federated framework and an adaptive local iteration count. The experimental findings affirm that our proposed methodology reduces the training energy outlay while maintaining caching efficacy.