Analyzing Big Data Originated from Data Communication Networks using K-Means Algorithm to Understand the Nature of Incoming Malicious Connections

Abstract

An environment is developed with a distributed Apache SPARK, deployed on Hadoop cluster for timely inference and classification of security incidents. Analysis of 85GB of network-packet dataset collected over four months is done (provided by CSIR-4PI, Govt. of India). K-means machine learning algorithm is used to analyze the network traffic based on various fields. By building and evaluating models, optimum number of clusters was determined. Clustering results are evaluated by calculating the clustering score using Within-Set Sum-of-Squared-Errors(WSSSE), entropy, Silhotte, Davies-Bouldin-Index and Dunn-Index. Several plots are visualized to understand the clustering analysis results and understand the nature of incoming malicious connections.