Non-blocking functional dependency discovery from data streams

Abstract

With the proliferation of sensors and IoT technologies, there is an increasing need to analyze information from data streams that they produce dynamically. However, the volume and velocity of this data require algorithms that mine knowledge as data are read from streams. The capability of dynamically extracting functional dependencies (fds) from data streams would not only permit to assess and improve the quality of data, but also provide knowledge on the evolution of data correlations within streams, allowing to understand the relevance that each feature has in predicting unknown features. In this paper, we propose a new discovery algorithm, namely COD3, which allows to continuous discovery fds holding on a data stream, as the data are read from it. COD3 represents the first proposal to use a non-blocking architectural model for discovering fds from data streams. Furthermore, we present novel data structures and a validation method to handle dynamic discovery and reduce data load inbound streams. Experimental evaluations demonstrate its effectiveness on both adapted real-world datasets and real data streams, such as those from air quality sensors. Moreover, by integrating COD3 with Bleach, a well-known fd-based data stream cleansing framework, we demonstrate its effectiveness in a real-world use case.