A Real-Time Data Processing based on Multilayer Windows and the Measurement Interchange Schema

Abstract

The dynamism of the economies requires updated data when decisions must be made. Currently, the Internet-of-Things incorporates a lot of different tiny and low-cost electronic devices able to be used for implementing a measurement process. However, those data sources are heterogeneous. The Data Stream Processing Architecture (DSPA) uses a Measurement and Evaluation (M&E) framework for warrantying the repeatability and consistency in the measurement process, jointly with the comparability of their results. The data arrive from the heterogeneous data sources to DSPA organized under the Measurement Interchange Schema (MIS), which is based on the project definition and allows matching each concept under monitoring with the corresponding measure from each device. Because the metric (or variable) related to each entity under monitoring should be analyzed in its own context and considering that data continuously arriving, a new strategy related to sliding logical windows is incorporated into the PAbMM_Win library, fostering processing alternatives jointly with the data interchange. Thus, the concepts of Layers and Carriers are introduced. On the one hand, the layers abstract the data processing from the logical viewpoint using the measurement project definition. On the other hand, the carriers allow transporting interpreted data to external data processing platforms in a transparent way. Its internal organization is here described. A discrete simulation shows its associated processing times. A snapshot on a whole measurement project could be taken and transported to a thirds' platform in only 16.72 ms with 1000 measures.