An empirical study on the performance overhead of code instrumentation in containerised microservices

Abstract

Code instrumentation is vital for analysing software behaviour and facilitating cloud computing observability and monitoring, especially in microservices and containers. Despite its benefits, instrumentation introduces complexity and performance overhead, which may inadvertently slow down systems and cause unexpected or erratic behaviour. In this study, we examine the effect of automated code instrumentation on the performance of containerised microservices by comparing instrumented systems against a baseline without instrumentation. Our experimental framework is based on key performance metrics, including response time, latency, throughput, and error percentage. It is executed using a rigorous methodology with a warm-up strategy to mitigate cold-start effects. Over 5000 experiments were conducted on 70 microservice APIs drawn from two open-source applications hosted on AWS and Azure to compare the results with baseline data. The experimental analysis comprises three stages: a pilot study on AWS, a case study on AWS and Azure, and an outlier analysis of the experimental results. Overall throughput decreased by up to 8.40 %, with some individual cases experiencing up to a 30 % reduction compared to the baseline, and response time and latency dropped by 20–49 %. Moreover, the results show more outlier cases in instrumentation results than in the baseline. Additionally, the results reveal more outlier cases in the instrumentation results compared to the baseline. The instrumentation has led to unexpected or erratic behaviour, as indicated by higher variations in response time, latency, and throughput values, along with increased error rates and occasional outlier values that were not observed in the non-instrumented run. This indicates that the performance differences we observed are attributable to overhead introduced by instrumentation, rather than inherent inefficiencies within the APIs themselves. Furthermore, statistical analysis utilised the Wilcoxon Signed-Rank test and mean ratios, with multiple approaches validating significant performance differences between instrumented and baseline conditions for both cloud services. A significance analysis using Cohen’s d indicates that the throughput and response time reductions in both platforms are not only statistically significant but also suggest considerable operational impact. These findings offer insights into automated code instrumentation's performance and impact on containerised microservices. It highlights the need to develop better and less impactful instrumentation techniques, and possibly towards the development of a new approach for large-scale software development and deployment in cloud environments that facilitates efficient instrumentation by design.