Heterogeneous Flow Scheduling using Deep Reinforcement Learning in Partially Observable NFV Environment

Abstract

Deep Reinforcement Learning (DRL) has yielded proficient controllers for complex tasks. DRL trains machine learning models for decision making to maximize rewards in uncertain environments such as network function virtualization (NFV). However, when facing limited information, agents often have difficulties making decisions at some decision point. In a real-world NFV environment, we may have incomplete information about network flow patterns. Compared with complete information feedback, it increases the difficulty to predict an optimal policy since important state information is missing. In this paper, we formulate a Partially Observable Markov Decision Process (POMDP) with a partially unknown NFV system. To address the shortcomings in real-world NFV, we conduct an extensive simulation to investigate the effects of adding recurrency to a Proximal Policy optimization (PPO2) by replacing the first post-convolutional fully-connected layer with a recurrent LSTM or adding stacked frames as input. The results show that RL based schedulers using stacking a history of frames in the PPO2’s input layer can easily adapt at evaluation time if the quality of observations changes.