A self-organizing resource allocation strategy based on Q-learning approach in ultra-dense networks

Abstract

In ultra-dense heterogeneous cellular networks, with the density of low power base stations (BSs) increasing, the inter-cell interference (ICI) can be extremely strong when all BSs reuse the same time-frequency resources. In this paper, after proving that allocating orthogonal (frequency) sub-bands to adjacent cells can perform better on throughput than reusing the whole bandwidth, we propose a multi-agent Q-learning based resources allocation (QLRA) approach as an enhanced solution to maximize the system performance. For the QLRA, we operate two learning paradigms: the distributed Q-learning (DQL) algorithm and the centralized Q-learning (CQL) algorithm. In the DQL scenario, all small cells learn independently without sharing any information. While in the CQL scenario, interaction between different agents is taken into consideration and resources are scheduled in a centralized way. Simulation results show that both QLRA scenarios can study an ideal resource allocation strategy automatically and achieve better performance on system throughput. Moreover, by scheduling resources in a centralized way, the CQL scenario can improve the system throughput furtherly.