Lyapunov-Guided Resource Allocation and Task Scheduling for Edge Computing Cognitive Radio Networks via Deep Reinforcement Learning

Abstract

Employing cognitive radio to facilitate edge computing is a promising solution to address the spectrum scarcity problem during massive task offloading. This article studies an edge computing cognitive radio network (EC-CRN), where multiple cognitive end devices (CEDs) offload tasks to multiple cognitive base stations (CBSs) for parallel edge computing over the licensed spectrum underlying primary users (PUs). To jointly optimize resource allocation and task scheduling, we formulate a long-term average system cost minimization (ASCM) problem subject to the constraints of end-edge task division, the maximum transmission power of CEDs, the peak interference power to PUs, the computing frequency of CBSs and the long-term energy consumption of CEDs. Due to the long-term objective and long-term constraint coupled slot-by-slot, we employ the Lyapunov optimization theory to derive the upper bound of the Lyapunov drift for the virtual energy consumption backlog and transform the original problem into a one-slot Lyapunov drift-plus-penalty minimization problem. Furthermore, we model the transformed problem by the Markov decision process (MDP) and propose the Lyapunov-guided resource allocation and task scheduling (LRATS) algorithm based on the deep reinforcement learning algorithm with proximal policy optimization (PPO), where the policy network is updated by the policy gradient ascent with adaptive trajectory expectation sampling, and the value network is updated by minimizing the mean squared error of temporal difference (TD). By comparing with benchmark algorithms based on greedy, particle swarm optimization (PSO), deep deterministic policy gradient (DDPG), twin delayed deep deterministic (TD3), and soft actor-critic (SAC) and making ablation experiments, we validate that the proposed algorithm can stably converge with a larger reward and effectively reduce the system cost.