An On-Line and Off-Line Pipeline-Based Architecture of the System for Gaps and Outlier Detection in Energy Data Stream

The quality of energy data (e.g. electric energy consumption, gas consumption data, energy production data) is very crucial issue in the energy domain. Low quality data is expressed in terms of large number of gaps and outliers in the data stream. These drawbacks can be caused by different reasons (e.g. devices faults, loss connections) but just-in-time detection of these cases is the mandatory step for further data handling. This paper describes an on-line and off-line pipeline-based architecture of a system for gaps and outlier detection in energy data streams. As decision making process is limited by time, it is proposed to split the data mining mechanism for gaps and outlier detection in real-time mode(on-line pipeline) from the adjustment mechanism of on-line pipeline's parameters (off-line pipeline). Each pipeline contains the sequence of filters for data handling. Filters in the on-line pipeline use only last input fraction of data. In contrast, filters in the off-line pipeline use all data stored in the data base. The results indicate that the proposed architecture allows to perform real-time gaps and outlier detection with the desired quality and constant latency despite increasing volume of data.