Research on stream processing engine and benchmarking framework

Abstract

Stream computing engine is an important part of big data system, and benchmarking is one of the main means to measure the engine's performance. In this paper, we compare the differences between two engines, Spark Streaming and Flink, in stream processing technologies. Then the open source benchmarking frameworks supporting stream processing and their respective characteristics are studied, and the HiBench testing framework is selected to test the two stream processing engines. The test results show that Flink is better than Spark Streaming in terms of performance in shuffle, stateful computation and windowed computation.