Automating anomaly detection for exploratory data analytics

Abstract

This paper discusses a design to automate the process of exploratory data analysis with an emphasis on outlier and anomaly detection. The paper discusses the domain of exploratory data analysis, the complexity involved in automating it and a solution leveraging the latest advances in computing to meet this. The solution details a framework that can accept data, understand the structure and type of variables, extract important variables and detect outliers or anomalies for understanding process bottlenecks. It takes advantage of big-data technologies and distributed computing (Hadoop and Spark) to make feasible the task of carrying out multiple lines of analysis and using intermediate results to drive analysis towards the desired goal. Statistical methods and visual data analytics form the core of the framework helping to automate exploratory data analysis, reducing time and focusing on the most valuable areas of concern in the data.