Using REO on ETL conceptual modelling: a first approach

Abstract

The formalization of software patterns has proven to be very useful in software developing, improving systems communication, data interchange across platforms, and simplifying the integration of processes and data flows. Populating a data warehouse (ETL) is often a very complex task demanding significant computational resources. It faces many drawbacks during its design and implementation, involving not only large volumes of data that must be processed but also undesirable change of business requirements. All of this leads frequently to reuse significant parts of other ETL implementations, adapting data structures and processes to comply with new requirements. Additionally, we believe that it's necessary a more simply and reliable approach for ETL conceptual modelling covering the "lack of mature" of this important part of ETL development. In this paper we explored a new approach to ETL conceptual modelling using the Reo coordination language, trying to evaluate its adequacy and expressiveness on the coordination of ETL tasks. A pattern-based approach was designed to map typical operations used in real world ETL scenarios from an initial Reo specification. For demonstration purposes, we present and discuss as two case studies, a slowly changing dimension and a surrogated key pipelining processes.