Optimize Critical Data Pattern Detection In Systems With Real Time Decisions

Abstract

The evolution of sensors, over a short period of time, just couple of decades, produced the rise and fast evolution of IoT (Internet of Things), from isolated "things" to "networks of things", storing data and using this data. Some of the common fields with IoT direct impact are industry, health, transportation, smart cities and smart buildings. The challenge on this new evolution is how to use the gathered data. Reports and data analysis are used from long time over big set of data and they will still remain an important purpose of this data. Recently, data received from sensors is used not only to trigger decision based on instant values but also to perform analysis on data stored and take decisions in real time, based on the data analysis. Because the data stored is huge and very diverse as format, various solutions to extract potentially dangerous data patterns in real time were studied. noSQL (non-relational database) and SQL (relational database) are used to store data in IoT systems. noSQL based solutions will store sensor data not depending on their type and format while a SQL one will keep the format with the main advantage of speed. This paper proposes two interconnected systems based on both databases: a noSQL for all data received from sensors, used later for reports and a SQL one with critical data, used to detect critical data patterns. The detection of critical data is performed on programmable gateway level and directed to the corresponding server. This paper discusses an architecture aiming to optimize critical pattern detection using jobs running on specific, customized, time interval. The time interval is chosen depending on data type. Simulation results are presented.