Design and Development of A Cloud-Based IDS using Apache Kafka and Spark Streaming

Abstract

Owing to the efficient resource management, accessibility, and high service availability, cloud computing has been leveraged by several intensive-data processing applications such as big data analytics, social media applications. These applications are typically based on the development of web service and web application. Even though web-based technology offers effective communication and implementation, it has been susceptible to various kinds of attack. In this paper, we investigate possible attacks on REST which is a commonly used protocol for the web service implementation. In REST, HTTP requests are mapped to GET, POST, PUT, and DELETE that have been proven to be prone to common attacks including Automated Brute Forcing on web-based login, HTTP flood attacks, SQL injections (SQLi), and Cross-Site Scripting (XSS). To this end, we propose a design and implementation of the cloud-based IDS to detect such attacks by employing Apache Kafka and Spark streaming to classify and process the high volume of user inputs in REST HTTP communication. To detect the anomalous inputs, we apply the signature-based approach to construct an IDS engine based on a set of known attack patterns that will be leveraged by the Spark Streaming. Specifically, we introduce a new string comparison collection that improves the False Positive (FP) rate in SQL injection detection, which has been a major issue in most proposed IDS currently available. In our experiment, the system is able to determine malicious patterns with high performance as well as to generate SMS alerts and log the event in a Google Cloud Storage Bucket in an efficient manner.