Kafka‐Shield: Kafka Streams‐based distributed detection scheme for IoT traffic‐based DDoS attacks

Abstract

With the rapid proliferation of insecure Internet of Things (IoT) devices, the security of Internet‐based applications and networks has become a prominent concern. One of the most significant security threats encountered in IoT environments is a Distributed Denial of Service (DDoS) attack. This attack can severely disrupt critical services and prevent smart devices from functioning normally, leading to severe consequences for businesses and individuals. It aims to overwhelm victims' resources, websites, and other services by flooding them with massive attack packets, making them inaccessible to legitimate users. Researchers have developed multiple detection schemes to detect DDoS attacks. As technology advances and other facilitating factors have increased, it is a challenge to identify such powerful attacks in real‐time. In this paper, we propose a novel distributed detection scheme for IoT network traffic‐based DDoS attacks by deploying it in a Kafka Streams processing framework named Kafka‐Shield. The Kafka‐Shield comprises two stages: design and deployment. Firstly, the detection scheme is designed on the Hadoop cluster employing a highly scalable H2O.ai machine learning platform. Secondly, a portable, scalable, and distributed detection scheme is deployed on the Kafka Streams processing framework. To analyze the incoming traffic data and categorize it into nine target classes in real time. Additionally, Kafka‐Shield stores each network flow with significant input features and the predicted outcome in the Hadoop Distributed File System (HDFS). It enables the development of new models or updating current ones. To validate the effectiveness of the Kafka‐Shield, we performed critical analysis using various configured attack scenarios. The experimental results affirm Kafka‐Shield's remarkable efficiency in detecting DDoS attacks. It has a detection rate of over 99% and can process 0.928 million traces in nearly 3.027 s.