Deep Reinforcement Learning Based Joint Uplink and Downlink Resource Allocation for URLLC

Ultra-reliable and low-latency communication (URLLC) is crucial for supporting mission-critical applications. Efficient resource allocation is an important issue in supporting URLLC in the 6G era. This paper investigates joint uplink and downlink resource allocation in a multiuser multiple-input single-output (MISO) orthogonal frequency-division multiple access (OFDMA) URLLC system in a finite blocklength regime. A subchannel assignment and time slot allocation problem is formulated with the objective to minimize the round-trip delay of URLLC under a reliability constraint, including the transmission delays and queuing delays of both uplink and downlink. Since the formulated resource allocation problem is a sequential decision making problem, a deep reinforcement learning (DRL) based proximal policy optimization (PPO) algorithm with a modified architecture is proposed to solve the problem. In the proposed algorithm, two actors are used to generate uplink and downlink resource allocation policies, respectively, and a shared critic is used to estimate the uplink and downlink state values while taking the states of both uplink and downlink into account. Meanwhile, a multi-head network structure, subchannel assignment masking, time slot allocation assistance and an adaptive hyperparameter are used for performance improvement. Simulation results demonstrate that the proposed URLLC resource allocation algorithm has a lower round-trip delay and performs better compared with benchmark algorithms.