Dynamic Resource Allocation for Real-Time Cloud XR Video Transmission: A Reinforcement Learning Approach

Abstract

The extend reality (XR) applications are increasing rapidly alongside the development of mobile Internet. Wireless resource allocation faces a significant challenge due to the high reliability and ultra-low latency characteristics of XR applications. So it is crucial to implement a rational resource allocation program. However, the complex characteristics of multi-user channels, coupled with the huge solution space of the resource allocation optimization problem, prevent conventional methods from efficiently and reliably deriving resource block (RB) allocation schemes. Therefore, in this paper, we construct a low-latency, highly dynamic cloud XR video transmission model considering the randomness of video arrival misalignment for different users, and we resort to newly developed deep reinforcement learning (DRL) techniques for solutions. To deal with the dimensional disaster problem with exponential order of RB allocation, we propose a parallel multi-DRL framework as the foundation for introducing two dynamic RB allocation algorithms: multi noisy double dueling deep Q networks (M-Noisy-D3QN) and multi soft actor critic (M-SAC). Both of the proposed algorithms can improve resource utilization and can achieve the exploration ability and complexity trade-off. Moreover, to address the challenge that RB allocation actions and system goals are not directly related, we design a novel reward function combining external rewards and internal incentives to establish a coherent connection between the two, i.e., solve the reward sparsity problem in DRL. Simulation results show that the proposed dynamic RB allocation methods can successfully serve nearly twice as many users as other benchmarks in case of bandwidth resource constraints.