Clustering Enabled Classification using Ensemble Feature Selection for Intrusion Detection

Abstract

Machine learning has been leveraged to increase the effectiveness of intrusion detection systems (IDSs). The focus of this approach, however, has largely be on detecting known attack patterns based on outdated datasets. In this paper, we propose an ensemble feature selection method along with an anomaly detection method that combines unsupervised and supervised machine learning techniques to classify network traffic to identify previously unseen attack patterns. To that end, three different feature selection techniques are used as part of an ensemble model that selects 8 common features. Moreover, k-Means clustering is used to first partition the training instances into k clusters using the Manhattan distance. A classification model is then built based on the resulting clusters, which represent a density region of normal or anomaly instances. This in turn helps determine the effectiveness of the clustering in detecting unknown attack patterns within the data. The performance of our classifier is evaluated using the Kyoto dataset, which was collected between 2006 and 2015. To our knowledge, no previous work proposed such a framework that combines unsupervised and supervised machine learning approaches using this dataset. Experimental results show the effectiveness of the proposed framework in detecting previously unseen attack patterns compared to the traditional classification approach.