Workflow Transformation for Real-Time Big Data Processing

Abstract

With the explosion of big data, processing and analyzing large numbers of continuous data streams in real-time, such as social media stream, sensor data streams, log streams, stock exchanges streams, etc., has become a crucial requirement for many scientific and industrial applications in recent years. Increased volume of streaming data as well as the demand for more complex real-time analytics require for execution of processing pipelines among heterogeneous event processing engines as a workflow. In this paper, we propose a workflow transformation for cost minimization in real-time big data processing on the heterogeneous systems. We first give the definition of stream-based workflow, and then we define eight different patterns as rules for workflow transformation, next, we give our workflow transformation algorithm based on our designed rules. Finally, our experiment shows that our proposed workflow transformation method can reduce the communication and computation cost effectively.