Small cell switch policy: A reinforcement learning approach

Abstract

Small cell is a flexible solution to satisfy the continuously increasing wireless traffic demand. In this paper, we focus on on-off switch operation on small cell base stations (SBS) in heterogeneous networks. In our scenario, the users can either choose SBS when it is active or macro cell base station (MBS) to transmit data. Start-up energy cost is considered when SBS switches on. The whole network acts as a queueing system, and network latency is also under consideration. The network traffic is modeled by a Markov Modulated Poisson Process (MMPP) whose parameters are unknown to the network control center. To maximize the system reward, we introduce a reinforcement learning approach to obtain the optimal on-off switch policy. The learning procedure is defined as a Markov Decision Process (MDP). An estimation method is proposed to measure the load of the network. A single-agent Q-learning algorithm is proposed afterwards. The convergence of this algorithm is proved. Simulation results are given to evaluate the performance of the proposed algorithm.