Deep Reinforcement Learning-Based Joint Caching and Routing in AI-Driven Networks

Abstract

To reduce redundant traffic transmission in both wired and wireless networks, optimal content placement problem naturally occurring in many applications is studied. In this paper, considering the limited cache capacity, unknown popularity distribution and non-stationary user demands, we address this problem by jointly optimizing content caching and routing with the objective of minimizing transmission cost. By optimizing the routing with the route-to-least cost-cache policy, the content caching process is modeled as a Markov decision process (MDP), aiming to maximize caching reward. However, the optimization problem consists of multiple nodes selecting caching contents, which leads to the combinatorial increase of the number of action dimensions with the number of possible actions. To handle this curse of dimensionality, we propose an intelligent caching algorithm by embedding action branching architecture into a dueling double deep Q-network (D3QN) to optimize caching decisions, and thus the agent at the controller can adaptively learn and track the underlying dynamics. Considering the independence of each branch, a marginal gain-based replacement rule is proposed to satisfy cache capacity constraint. Our simulation results show that compared with the prior art, the caching reward and hit rate of the proposed algorithm are increased by 35.3% and 33.6% respectively on average.