Robust Defensive Cyber Agent for Multi-Adversary Defense

Abstract

Modern cyber environments are becoming increasingly complex and distributed, often organized into multiple interconnected subnets and nodes. Even relatively small-scale networks can exhibit significant security challenges due to their dynamic topologies and the diversity of potential attack vectors. In modern cyber environments, human-led defense alone is insufficient due to delayed response times, cognitive overload, and limited availability of skilled personnel, particularly in remote or resource-constrained settings. These challenges are intensified by the growing diversity of cyber threats, including adaptive and machine learning-based attacks, which demand rapid and intelligent responses. Addressing this, we propose a reinforcement learning (RL)-based framework that integrates eXtreme Gradient Boosting (XGBoost) and transformer architectures to develop robust, generalizable defensive agents. The proposed agents are evaluated against both baseline defenders trained to counter specific adversaries and hierarchical generic agents representing the current state-of-the-art. Experimental results demonstrate that the RL-XGBoost (integration of RL and XGBoost) agent consistently achieves superior performance in terms of defense accuracy and efficiency across varied adversarial strategies and network configurations. Notably, in scenarios involving changes to network topology, both RL-Transformer (RL combined with transformer architectures) and RL-XGBoost agents exhibit strong adaptability and resilience, outperforming specialized blue agents and hierarchical agents in performance consistency. In particular, the RL-Transformer variant (RL-BERT) demonstrates exceptional robustness when attacker entry points are altered, effectively capturing long-range dependencies and temporal patterns through its self-attention mechanism. Overall, these findings highlight the RL-XGBoost model’s potential as a scalable and intelligent solution for multi-adversary defense in dynamic and heterogeneous cyber environments.