Intelligent preamble allocation for coexistence of mMTC/URLLC devices: A hierarchical Q-learning based approach

The emergence of various commercial and industrial Internet of Things (IoT) devices has brought great convenience to people's life and production. Both low-power, massively connected mMTC devices (MDs) and highly reliable, low-latency URLLC devices (UDs) play an important role in different application scenarios. However, when dense MDs and UDs periodically initiate random access (RA) to connect the base station and send data, due to the limited preamble resources, preamble collisions are likely to occur, resulting in device access failure and data transmission delay. At the same time, due to the high-reliability demands of UDs, which require smooth access and fast data transmission, it is necessary to reduce the failure rate of their RA process. To this end, we propose an intelligent preamble allocation scheme, which uses hierarchical reinforcement learning to partition the UD exclusive preamble resource pool at the base station side and perform preamble selection within each RA slot at the device side. In particular, considering the limited processing capacity and energy of IoT devices, we adopt the lightweight Q-learning algorithm on the device side and design simple states and actions for them. Experimental results show that the proposed intelligent scheme can significantly reduce the transmission failure rate of UDs and improve the overall access success rate of devices.