Towards comparison of real time stream processing engines

Abstract

The real time stream processing engines are developed for different specific use cases, which incorporates various domains such as, IOT, finance, advertisement, telecommunications, healthcare etc. These stream processing engines are based on distributed processing models, where unbounded data streams are processed. Semantics of data stream is determined after complete scanning of whole data sets, which becomes inconvenient in real time stream processing to process entire data stream at once. Windowing mechanisms are used for processing data stream in a predefine topology with fixed number of operations such as, join, aggregate, filter etc. In this paper, a comparative study is performed with existing stream processing engines. This comparison provides a direction for choosing an appropriate stream processing engine. A modified master-slave model for stream processing is discussed for reducing latency, improving scalability and fault tolerance.