Adaptive traffic engineering with segment routing through deep reinforcement learning

Abstract

Segment Routing (SR) is a source routing technique that has been widely used in Traffic Engineering (TE) because of its scalability and flexibility. Despite extensive research on Traffic Engineering with Segment Routing (SR-TE) in recent years, online SR-TE still encounters challenges such as the absence of real-time traffic matrices (TMs), slow online decision speed, and unsatisfactory TE performance. Although TE with Reinforcement Learning (RL) may obviate the need for real-time TMs in online TE, existing studies struggle to handle the vast number of candidate routing plans introduced by SR-TE, as well as have significant training overhead. In this paper, we propose an online adaptive SR-TE algorithm named Adpt-SRTE. With the help of deep reinforcement learning (DRL), Adpt-SRTE is first trained with pre-collected historical TMs, and then provides SR routing configuration for new TMs online when only real-time link utilization is known. To deal with the massive number of candidate routing plans, Adpt-SRTE strategically combines the Proximal Policy Optimization (PPO) algorithm with action branching architecture. Besides, appropriate training methods are used to improve TE performance and reduce training overhead. Experimental results demonstrate that Adpt-SRTE can achieve good TE performance for both short and long time scale up to weeks, reducing the maximum link utilization by up to 33%. Besides, it has low offline training overhead, short online decision time and low path configuration overhead.