Non-intrusive Monitoring of Stream Processing Applications

Abstract

Stream processing applications have emerged as a popular way for implementing high-volume data processing tasks. In contrast to traditional data processing models that persist data to databases and then execute queries on the stored data, stream processing applications continuously execute complex queries on incoming data to produce timely results in reaction to events observed in the processed data. To cope with the request load, components of a stream processing application are usually distributed across multiple machines. In this context, performance monitoring and testing are naturally important for stakeholders to understand as well as analyze the runtime characteristics of deployed applications to identify issues and inform decisions. Existing approaches for monitoring the performance of distributed systems, however, do not provide sufficient support for targeted monitoring of stream processing applications, and require changes to the application code to enable the integration of application-specific monitoring data. In this paper we present MOSAIC, a service oriented framework that allows for in-depth analysis of stream processing applications by non-intrusively adding functionality for acquiring and publishing performance measurements at runtime, to the application. Furthermore, MOSAIC provides a flexible mechanism for integrating different stream processing frameworks, which can be used for executing and monitoring applications independent from a specific operator model. Additionally, our framework provides an extensible approach for gathering and analyzing measurement data. In order to evaluate our solution, we developed a scenario application, which we used for testing and monitoring its performance on different stream processing engines.