CAT: content-aware tracing and analysis for distributed systems

Tracing and analyzing the interactions and exchanges between nodes is fundamental to uncover performance, correctness and dependability issues almost unavoidable in any complex distributed system. Existing monitoring tools acknowledge this importance but, so far, restrict tracing to the external attributes of I/O messages, thus missing a wealth of information in them. We present CaT, a non-intrusive content-aware tracing and analysis framework that, through a novel similarity-based approach, is able to comprehensively trace and correlate the flow of network and storage requests from applications. By supporting multiple tracing tools, CaT can balance the coverage of captured events with the impact on applications' performance. The conducted experimental evaluation considering two widely used applications (TensorFlow and Apache Hadoop) shows how CaT can improve the analysis of distributed systems. The results also exemplify the trade-offs that can be used to balance tracing coverage and performance impact. Interestingly, in certain cases, full coverage of events can be attained with negligible performance and storage overhead.