M-TOP: multi-target operator placement of query graphs for data streams

Abstract

Nowadays, many applications processes stream-based data, such as financial market analysis, network intrusion detection, or visualization applications. To process stream-based data in an application-independent manner, distributed stream processing systems emerged. They typically translate a query to an operator graph, place the operators to stream processing nodes, and execute them to process the streamed data. The operator placement is crucial in such systems, as it deeply influences query execution. Often, different stream-based applications require dedicated placement of query graphs according to their specific objectives, e.g. bandwidth not less than 500 MBit/s and costs not more that 1 cost unit. This fact constraints operator placement. Existing approaches do not take into account application-specific objectives, thus not reflecting application-specific placement decisions. As objectives might conflict among each other, operator placement is subject to delicate trade-offs, such as bandwidth maximization is more important than cost reduction. Thus, the challenge is to find a solution which considers the application-specific objectives and their trade-offs. We present M-TOP, an QoS-aware multi-target operator placement framework for data stream systems. Particularly, we propose an operator placement strategy considering application-specific targets consisting of objectives, their respective trade-offs specifications, bottleneck conditions, and ranking schemes to compute a suitable placement. We integrated M-TOP into NexusDS, our distributed data stream processing middleware, and provide an experimental evaluation to show the effectiveness of M-TOP.