RAGN: Detecting unknown malicious network traffic using a robust adaptive graph neural network

Abstract

As network environments evolve, detecting unknown malicious network traffic becomes increasingly challenging due to the dynamic and sophisticated nature of modern cyberattacks. Graph Attention Networks (GATs) have shown promise in modeling complex network interactions but remain vulnerable to adversarial attacks that exploit weaknesses in the graph structure. In this work, we propose the Robust Adaptive Graph Neural Network (RAGN), an enhanced GAT-based framework that introduces adaptive attention mechanisms to improve detection accuracy and robustness against adversarial manipulations in network traffic graphs. RAGN iteratively adjusts the graph structure and feature space to suppress adversarial perturbations by assigning lower attention scores to unreliable edges and refining feature representations based on the feature smoothness regularization principle. To assess the robustness of the proposed RAGN model and compare it with baseline models, we introduced an effective dynamic graph attack method known as Semantic-Preserving Adversarial Node Injection (SPAN). We benchmarked its performance against state-of-the-art graph attack methods, including DICE, DGA, and RWCS. SPAN incrementally injects small batches of malicious nodes, refining their edges and features to target both the structural and temporal aspects of dynamic graphs. It preserves semantic integrity, and generates effective yet imperceptible perturbations, providing a rigorous test of the resilience of graph neural networks. Experiments conducted on four datasets, demonstrate that RAGN demonstrates robustness against adversarial, and zero-day attacks. It also demonstrates resilience against targeted, malicious node injection attacks in dynamic network environments. RAGN demonstrated consistent robustness, with misclassification rates increasing only marginally (by less than 1.2%) even under significant dynamic changes.