Data integration in scalable data analytics platform for process industries

Abstract

The main objective of work presented in this paper is to introduce the architectural overview of the big data analytics platform for support of process industries. Our aim was to design and develop the cross-sectorial scalable environment, which will enable the data collection from different sources and support the development of predictive functions to help the process industries in optimizing of their production processes. This paper introduces the components of Big Data Storage and Analytics platform which is the core component of the developed cross-sectorial environment. Currently, it is built on top of the Apache Hadoop technology stack and relies on Hadoop distributed file system. On the other hand, we present the idea of integration of the data obtained from different production environments. Data integration is implemented using the Apache Nifi and we designed the workflows for processing both interval and real-time data from the production sites. In this case, we consider two pilot cases, an aluminium factory in France and a plastic molding factory in Portugal.