Data-Driven Monitoring for Cloud Compute Systems

The end-to-end monitoring of inter-dependent applications in the cloud is challenging. Difficulties arise from the complexity of computations and the highly distributed nature of the deployment. Due to the lack of a comprehensive observability solution, it is very difficult to apply autonomous mechanisms to ensure service guarantees in the cloud. To tackle the problem, we propose the method of data-driven monitoring, that provides a detailed, live view on how data is flowing through a possibly complex compute system. The method is based on the tracing of individual input events and the collection of resource usage metrics along the paths. By reconstructing causal and temporal relationships, we can detect degradations in performance, pinpoint root causes and apply corrective actions before end-to-end requirements are endangered. To demonstrate the potential of the concept, we created a prototype implementation in a big data compute platform, and also developed two automated optimization algorithms.