CNDAS-WF: Cloud Native Data Analysis System Based On Workflow Engine

Abstract

With the development of modern big data technology, data size in daily life is expanding rapidly and data relationship is more complex. However, the requirements of data analysis for different resources continuous to surging. Therefore, how to handle a large number of data analysis tasks with complex dependencies efficiently become the challenge. In this paper, we design and implement a cloud native data analysis system based on workflow engine. The system arranges the data analysis tasks, which deployed by containers, with dependency through the workflow engine based on cloud native technology. Flexibility of container cloud makes data analysis procedure effective and efficient. In addition, we designed a workflow engine and an operation and maintenance subsystem for overall system platform anomaly detection. Finally, we verify the effectiveness and efficiency of the system through scientific workflow data. The cloud native data analysis system based on workflow engine has passed all tests and has been applied in small and medium-sized enterprises.