Big data analysis architecture for multi IDS sensors using memory based processor

Abstract

The massive internet usage is followed by the rise of cyber-related crime such as information stealing, denial-of-service (DoS) attack, trojan and malware. To cope with the threats, one of most popular choice is using Intrusion Detection System (IDS). The logs produced by IDS in a day is huge and the limitation of computing power is the main problem to process that logs files. In this paper, we propose a big data analysis architecture of multi IDS sensors using in-memory data processing. Deployed IDS sensors are taking an extra role as computation slave to build scalable data analysis platform for network security analysis. So, adding more sensors means expanding computational resources. Adding to three sensors are helping data computation of clustering algorithm faster up to 27% comparing to the computation by using only one sensor. This research also introduces the use of memory-based processor, this system provides 7,9 times faster data processing than conservative MapReduce operation. And moreover, we also have performed botnets classification over Spark RDD that give high accuracy result to 99%.