A “Fast Data” architecture: Dashboard for anomalous traffic analysis in data networks

Abstract

Fast Data is a new Big Data computing paradigm that ensures requirements such as Real-Time processing of continuous data stream, storage at high rates and low latency with no data losses. In this work we propose a “Fast Data” architecture for a specific kind of software application in which input data arrive very fast and the results for each processed data have to match such input rates. We applied this architecture to build a Dashboard for Anomalous Traffic Analysis in Data Networks. In order to fulfill the requirements of Real-Time processing and no data losses, we carry out a design that consists of a pattern of dynamic tree of process pipelines, where the number of branches increases proportionally to the input data rate. Two different approaches have been followed to implement this design pattern: one based in a well-known set of products from the Big Data ecosystem; and the other built with Kafka, Zookeeper and a set of components designed and implemented by us. These two implementations have been compared in terms of velocity and scalability performance. As a result, the implementation built with our own components is significantly faster and scalable than the traditional one. The good results obtained by using both the design pattern of dynamic tree of process pipelines and our implementation make them very suitable for its use in other scenarios and applications such as smart cities, environment monitoring, industry 4.0, distributed control systems, etc.