A Curriculum Framework for Autonomous Network Defense using Multi-agent Reinforcement Learning

Early threat detection is an increasing part of the cybersecurity landscape given the growing scale and scope of cyberattacks in the recent years. Increasing exploitation of software vulnerabilities, especially in the manufacturing sector, demonstrates the ongoing need for autonomous network defense. In this work, we model the problem as a zero-sum Markov game between an attacker and defender reinforcement learning agents. Previous methods test their approach on a single topology or limit the agents to a subset of the network. However, real world networks are rarely fixed and often add or remove hosts based on demand, link failures, outages, or other factors. We consider two types of topologies: static topologies that remain fixed throughout training and a dynamic topology curriculum. The proposed robust training curriculum incorporates network topologies to build more general, capable agents. We also use Proximal Policy optimization (PPO) which offers a good balance of computational complexity and convergence speed. We evaluate various threat scenarios in terms of the exploitability and impact and conclude that the curriculum improves the defender’s win rate over training on a static topology by exposing the agent to more challenging environments over time.