Towards a Framework for Data Pipeline Discovery

Abstract

With the recent developments of Internet of Things (IoT) and cloudbased technologies, massive amounts of data are generated by heterogeneous sources and stored through dedicated cloud solutions. Often organizations generate much more data than they are able to interpret, and current Cloud Computing technologies cannot fully meet the requirements of the Big Data processing applications and their data transfer overheads [3]. Many data are stored for compliance purposes only but not turned into value, thus becoming Dark Data, which are not only an unused value but also pose a risk for organizations [7, 18].