Enabling A Load Adaptive Distributed Stream Processing Platform on Synchronized Clusters

Abstract

Distributed stream processing (DSP) platforms enable simplified development of applications that can process continuous unbounded streams of data at a high speed. Leveraging large scale cluster management frameworks, DSP can scale to analyze data in real-time with different types of operators, each running on a cluster node. The scalability and resource utilization depend on the allocation of operators on clusters. Since the data volume and rate can be unpredictable, static mapping between operators and cluster resources results in unbalanced operator load distribution. This paper proposes a software layer that is load-adaptive between a DSP platform and clusters. It allows dynamic transferring of an operator to different cluster nodes at runtime and keeps the process transparent to developers. We present a prototype implemented on Yahoo's S4. Our implementation is evaluated by a top-N topic list application on Twitter streams. The results demonstrate improved stream processing throughputs and cluster resource utilization.