A PaaS based metadata-driven ETL framework

Abstract

Knowledge discovery has often used as a background application to motivate many technical problems in ETL research. However, traditional ETL tools face new challenges include tremendous amount of data and limitation of computing ability and so on. Meanwhile, MapReduce parallel computing model has been widely used in recent years. In This paper, we first analyze the problems of existing ETL tools and propose a metadata-driven ETL service model, and then summarize the types of metadata and their application scopes. Based on this metadata-driven ETL service model, we put forward a concrete ETL framework combined ETL with MapReduce algorithm framework and provided as PaaS to meet the requirements. Afterwards, many significant services are also discussed. At last, we illustrate some strategies for advancing the flexibility, extensibility of the framework and promote the reusability of ETL components and ETL application. In conclusion, practices have proved that the model and the framework proposed in this paper have advantages that open-source or commercial ETL tools do not have and can deal the problem of processing large scale data.