Design and application of real-time network abnormal traffic detection system based on Spark Streaming

Abstract

In order to realise the rapid analysis and identification of abnormal traffic in real-time networks, a distributed real-time network abnormal traffic detection system (DRNATDS) was designed, which could effectively analyse abnormal network traffic. DRNATDS provided effective real-time big data analysis platform and guaranteed network security. The paper proposes K-means algorithm based on relative density and distance, integrated with Spark Streaming and Kafka. It could effectively detect various network attacks under real-time data stream. The experimental results show that DRNATDS has good high availability and stability. Compared to other algorithms, K-means algorithm based on relative density and distance could more effectively identify abnormal network traffic and improve the recognition rate.