Flow: A Stream Processing System Simulator

Abstract

Stream processing is an important emerging computational model for performing complex operations on and across multi-source, high volume, unpredictable dataflows. We present Flow, a platform for parallel and distributed stream processing system simulation that provides a flexible modeling environment for analyzing stream processing applications. The Flow stream processing system simulator is a high performance, scalable simulator that automatically parallelizes chunks of the model space and incurs near zero synchronization overhead for stream application graphs that exhibit feed-forward behavior. We show promising multi-threaded and multi-process event rates exceeding 80 million events per second on a cluster with 256 processor cores.