A robust anomaly detection system

Abstract

Data mining techniques helps to sift through large amount of data for patterns and characteristic rules. Due to great possibility of malicious data entering in any field of concern, it has become a necessity to build not just a generalized model for anomaly detection but also train the same model to work with utmost precision. K-means clustering algorithm although is one of the most easiest and quite popular unsupervised clustering algorithm, it can be used to dovetail PCA and Robust MCD to build a very generalized and robust anomaly detection system. Standard problems resulting from K-means algorithm is its constant attempt to find local minima and result in a cluster that leads to ambiguity, however if the same K-means algorithm is combined with principal component analysis technique(PCA),it results in the formation of more closely centered cluster that works well with K-means algorithm, and with the application of a customized robust and adaptive outlier detection algorithm can provide a great boost to the the anomaly detection problem. The system proposed is a robust anomaly detection system that can be applied to any field from health to networks in order to accurately detect outliers due to the robust and adaptive nature of MCD algorithm developed in this paper.