Deep Reinforcement Learning-Based Dual-Timescale Service Caching and Computation Offloading for Multi-UAV Assisted MEC Systems

Abstract

The emergence of unmanned aerial vehicles (UAVs) ushers in a new era for mobile edge computing (MEC), significantly expanding its range of service and potential applications. Due to the limited storage capacity and energy budget of UAVs, it is crucial to determine a reasonable service caching and task offloading strategy. Service caching means that task-related programs and the associated databases are cached on edge servers. In this paper, we consider the time latency and energy consumption caused by frequent changes to the service caching, aiming to jointly optimize the computational offloading, resource allocation, and service caching in multi-UAV assisted MEC systems at different time scales. The objective of this optimization is to reduce the overall system delay while staying within the energy limitations of both the UAVs and ground devices. An improved service caching policy (SCP) is proposed, which is based on task popularity and utilizes the greedy dual size frequency (GDSF) algorithm. The SCP is combined with the twin delayed deep deterministic policy gradient (TD3) algorithm to propose an innovative dual timescale TD3 (DTTD3) algorithm. The numerical outcomes obtained from a substantial number of simulation experiments demonstrate that DTTD3 outperforms existing benchmark methods in terms of convergence and parameter optimization.