SYN Flood Attack Detection and Mitigation using Machine Learning Traffic Classification and Programmable Data Plane Filtering

Abstract

Distributed Denial of Service (DDoS) attacks are widely used by malicious actors to disrupt network infrastructures/services. A common attack is TCP SYN Flood that attempts to exhaust memory and processing resources. Typical mitigation mechanisms, i.e. SYN cookies require significant processing resources and generate large rates of backscatter traffic to block them. In this paper, we propose a detection and mitigation schema that focuses on generating and optimizing signature-based rules. To that end, network traffic is monitored and appropriate packet-level data are processed to form signatures i.e. unique combinations of packet field values. These are fed to machine learning models that classify them to malicious/benign. Malicious signatures corresponding to specific destinations identify potential victims. TCP traffic to victims is redirected to high-performance programmable XDPenabled firewalls that filter off ending traffic according to signatures classified as malicious. To enhance mitigation performance malicious signatures are subjected to a reduction process, formulated as a multi-objective optimization problem. Minimization objectives are (i) the number of malicious signatures and (ii) collateral damage on benign traffic. We evaluate our approach in terms of detection accuracy and packet filtering performance employing traces from production environments and high rate generated attack traffic. We showcase that our approach achieves high detection accuracy, significantly reduces the number of filtering rules and outperforms the SYN cookies mechanism in high-speed traffic scenarios.