Proceedings of the ACM/SPEC International Conference on Performance Engineering最新文献

{"title":"Courier: Real-Time Optimal Batch Size Prediction for Latency SLOs in BigDL","authors":"Diego Albo Martínez, Sharwin Bobde, Tomasz Motyka, L. Chen","doi":"10.1145/3427921.3450233","DOIUrl":"https://doi.org/10.1145/3427921.3450233","url":null,"abstract":"Distributed machine learning has seen immense rise in popularity in recent years. Many companies and universities are utilizing computational clusters to train and run machine learning models. Unfortunately, operating such a cluster imposes large costs. It is therefore crucial to attain as high system utilization as possible. Moreover, those who offer computational clusters as a service, apart from keeping high utilization, also have to meet the required Service Level Agreements (SLAs) for the system response time. This becomes increasingly more complex in multitenant scenarios, where the time dedicated to each task has to be limited to achieve fairness. In this work, we analyze how different parameters of the machine learning job influence the response time as well as system utilization and propose Courier. Courier is a model that, based on the type of machine learning job, can select a batch size such that the response time adheres to the Service Level Objectives (SLOs) specified, while also rendering the highest possible accuracy. We gather the data by conducting real-world experiments on a BigDL cluster. Later on, we study the influence of the factors and build several predictive models which lead us to the proposed Courier model.","PeriodicalId":433195,"journal":{"name":"Proceedings of the ACM/SPEC International Conference on Performance Engineering","volume":"12 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132225246","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"RENOIR: Accelerating Blockchain Validation using State Caching","authors":"Nitin Awathare, Sourav Das, V. Ribeiro, U. Bellur","doi":"10.1145/3427921.3450247","DOIUrl":"https://doi.org/10.1145/3427921.3450247","url":null,"abstract":"A Blockchain system such as Ethereum is a peer to peer network where each node works in three phases: creation, mining, and validation phases. In the creation phase, it executes a subset of locally cached transactions to form a new block. In the mining phase, the node solves a cryptographic puzzle (Proof of Work-PoW) on the block it forms. On receiving a block from another peer, it starts the validation phase, where it executes the transactions in the received block in order to ensure all transactions are valid. This execution also updates the blockchain state, which must be completed before creating the next block. A long block validation time lowers the system's overall throughput and brings the well known Verifier's dilemma into play. Additionally, this leads to wasted mining power utilization (MPU). Through extensive measurement of 2000 nodes from the production Ethereum network we find that during block validation, nodes redundantly execute more than 80% of the transactions in greater than 75% of the blocks they receive - this points to significant potential to save time and computation during block validation. Motivated by this, we present RENOIR, a novel mechanism that caches state from transaction execution during the block creation phase and reuses it to enable nodes to skip (re)executing these transactions during block validation. Our detailed evaluation of RENOIR on a 50 node testbed mimicking the top 50 Ethereum miners illustrates that when gas limit is increased to 20 times the default value, to accommodate computationally intensive transactions, RENOIR reduces validation time by 90% compared to Ethereum. In addition, throughput of Ethereum reduces from 35326 tx/hour to 24716 tx/hour and MPU from 96% to 67% but these barely change for RENOIR. Furthermore, we deploy a node running RENOIR on the production Ethereum network. Our measurement illustrates that RENOIR reduces the block validation time by as much as 50%.","PeriodicalId":433195,"journal":{"name":"Proceedings of the ACM/SPEC International Conference on Performance Engineering","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131154090","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Learning Queuing Networks via Linear Optimization","authors":"Emilio Incerto, Annalisa Napolitano, M. Tribastone","doi":"10.1145/3427921.3450245","DOIUrl":"https://doi.org/10.1145/3427921.3450245","url":null,"abstract":"The automatic derivation of analytical performance models is an essential tool to promote a wider adoption of performance engineering techniques in practice. Unfortunately, despite the importance of such techniques, the attempts pursuing that goal in the literature either focus on the estimation of service demand parameters only or suffer from scalability issues and sub-optimality due to the intrinsic complexity of the underlying optimization methods. In this paper, we propose an efficient linear programming approach that allows to derive queuing network (QN) models from sampled execution traces. For doing so, we rely on a deterministic approximation of the average dynamic of QNs in terms of a compact system of ordinary differential equations. We encode these equations into a linear optimization problem whose decision variables can be directly related to the unknown QN parameters, i.e., service demands and routing probabilities. Using models of increasing complexity, we show the efficiency and the effectiveness of our technique in yielding models with high prediction power.","PeriodicalId":433195,"journal":{"name":"Proceedings of the ACM/SPEC International Conference on Performance Engineering","volume":"25 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116661694","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"ConfProf: White-Box Performance Profiling of Configuration Options","authors":"Xue Han, Tingting Yu, Michael Pradel","doi":"10.1145/3427921.3450255","DOIUrl":"https://doi.org/10.1145/3427921.3450255","url":null,"abstract":"Modern software systems are highly customizable through configuration options. The sheer size of the configuration space makes it challenging to understand the performance influence of individual configuration options and their interactions under a specific usage scenario. Software with poor performance may lead to low system throughput and long response time. This paper presents ConfProf, a white-box performance profiling technique with a focus on configuration options. ConfProf helps developers understand how configuration options and their interactions influence the performance of a software system. The approach combines dynamic program analysis, machine learning, and feedback-directed configuration sampling to profile the program execution and analyze the performance influence of configuration options. Compared to existing approaches, ConfProf uses a white-box approach combined with machine learning to rank performance-influencing configuration options from execution traces. We evaluate the approach with 13 scenarios of four real-world, highly-configurable software systems. The results show that ConfProf ranks performance-influencing configuration options with high accuracy and outperform a state of the art technique.","PeriodicalId":433195,"journal":{"name":"Proceedings of the ACM/SPEC International Conference on Performance Engineering","volume":"57 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130838558","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"PECS'21: The First Workshop on Performance and Energy-efficiency of Concurrent Systems","authors":"Romolo Marotta, F. Quaglia","doi":"10.1145/3427921.3450229","DOIUrl":"https://doi.org/10.1145/3427921.3450229","url":null,"abstract":"Concurrent systems, based on (distributed) multi/many-core processing units, are the nowadays reference computing architecture. The (continuously-growing) level of hardware parallelism they offer has led these platforms to play a central role at any scale, ranging from data centers, to personal (mobile) devices. Optimizing performance and/or ensuring energy efficiency when running complex software stacks on top of these systems is extremely challenging due to several aspects, like data dependencies or resource sharing (and interference) among application threads, as well as VMs. Furthermore, hardware accelerators like GPGPUs or FPGAs introduce a level of heterogeneity that can potentially offer further opportunities for combined gain in performance and energy efficiency, if correctly exploited. The goal of this workshop is to establish a venue for both academia and industry experts and practitioners, where they can discuss challenges, perspectives and opportunities given by researching on scalable, energy-efficient and secure software deployed on top of modern (heterogeneous) concurrent platforms.","PeriodicalId":433195,"journal":{"name":"Proceedings of the ACM/SPEC International Conference on Performance Engineering","volume":"54 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131067925","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Libra: A Benchmark for Time Series Forecasting Methods","authors":"A. Bauer, Marwin Züfle, Simon Eismann, Johannes Grohmann, N. Herbst, Samuel Kounev","doi":"10.1145/3427921.3450241","DOIUrl":"https://doi.org/10.1145/3427921.3450241","url":null,"abstract":"In many areas of decision making, forecasting is an essential pillar. Consequently, there are many different forecasting methods. According to the \"No-Free-Lunch Theorem\", there is no single forecasting method that performs best for all time series. In other words, each method has its advantages and disadvantages depending on the specific use case. Therefore, the choice of the forecasting method remains a mandatory expert task. However, expert knowledge cannot be fully automated. To establish a level playing field for evaluating the performance of time series forecasting methods in a broad setting, we propose Libra, a forecasting benchmark that automatically evaluates and ranks forecasting methods based on their performance in a diverse set of evaluation scenarios. The benchmark comprises four different use cases, each covering 100 heterogeneous time series taken from different domains. The data set was assembled from publicly available time series and was designed to exhibit much higher diversity than existing forecasting competitions. Based on this benchmark, we perform a comprehensive evaluation to compare different existing time series forecasting methods.","PeriodicalId":433195,"journal":{"name":"Proceedings of the ACM/SPEC International Conference on Performance Engineering","volume":"56 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122881772","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"An Exploratory Study of the Impact of Parameterization on JMH Measurement Results in Open-Source Projects","authors":"H. Samoaa, P. Leitner","doi":"10.1145/3427921.3450243","DOIUrl":"https://doi.org/10.1145/3427921.3450243","url":null,"abstract":"The Java Microbenchmarking Harness (JMH) is a widely used tool for testing performance-critical code on a low level. One of the key features of JMH is the support for user-defined parameters, which allows executing the same benchmark with different workloads. However, a benchmark configured with n parameters with m different values each requires JMH to execute the benchmark mn times (once for each combination of configured parameter values). Consequently, even fairly modest parameterization leads to a combinatorial explosion of benchmarks that have to be executed, hence dramatically increasing execution time. However, so far no research has investigated how this type of parameterization is used in practice, and how important different parameters are to benchmarking results. In this paper, we statistically study how strongly different user parameters impact benchmark measurements for 126 JMH benchmarks from five well-known open source projects. We show that 40% of the studied metric parameters have no correlation with the resulting measurement, i.e., testing with different values in these parameters does not lead to any insights. If there is a correlation, it is often strongly predictable following a power law, linear, or step function curve. Our results provide a first understanding of practical usage of user-defined JMH parameters, and how they correlate with the measurements produced by benchmarks. We further show that a machine learning model based on Random Forest ensembles can be used to predict the measured performance of an untested metric parameter value with an accuracy of 93% or higher for all but one benchmark class, demonstrating that given sufficient training data JMH performance test results for different parameterizations are highly predictable.","PeriodicalId":433195,"journal":{"name":"Proceedings of the ACM/SPEC International Conference on Performance Engineering","volume":"228 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117051343","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Welcome to the 3rd Workshop on Education and Practice of Performance Engineering","authors":"Alberto Avritzer, K. Trivedi, A. Iosup","doi":"10.1145/3427921.3450225","DOIUrl":"https://doi.org/10.1145/3427921.3450225","url":null,"abstract":"The Workshop on Education and Practice of Performance Engineering, in its 3rd edition, brings together University researchers and Industry Performance Engineers to share education and practice experiences. The recommendations from previous WEPPE workshops pointed to the need to form a team of critical thinkers that also have good communication skills. This workshop is a forum to share experiences between researchers that are actively teaching performance engineering and of Performance Engineers that are applying Performance Engineering techniques in industry. Specifically, as ICPE 2021 is virtual, for the 3rd edition, we will give special attention to online Q/A interactions supported by off-line videos. To achieve these goals we have put together a very exciting program, consisting of 3 full papers and 5 invited talks. We start by discussing recommendations from previous WEPPE workshops with the talk on Minding the Gap between Education and Practice by Alberto Avritzer, Kishor Trivedi, Alexandru Iosup, where the co-chairs will provide a summary report of previous WEPPE workshops as a starting point for WEPPE 2021 discussion. We continue with a presentation on \"The role of analytical models in the engineering and science of computer systems\", where Y.C Tay, National University of Singapore, reviews the role of analytical performance modeling in engineering computer systems and developing a science of computer systems. In the talk on \"Performance monitoring guidelines\" by Maria Calzarossa, Luisa Massari, Daniele Tessera, Università di Pavia and Università Cattolica del Sacro Cuore, the authors present the topic of Performance Monitoring, which is the process for collecting measurements on infrastructures and services. The talk on \"Experience with Teaching Performance Measurement and Testing\" by Andre Bondi, Razieh Saremi, Software Performance and Scalability Consulting LLC and Stevens Institute of Technology, the authors review a Course on Functional Testing that they have developed and taught where performance engineering aspects were introduced. In the talk \"An Analysis of Distributed Systems Syllabi With a Focus on Performance-Related Topics\" by Cristina Abad, Alexandru Iosup, Edwin Boza, Eduardo Ortiz-Holguin, Escuela Superior Politecnica del Litoral and Vrije Universiteit Amsterdam, the authors analyze a large set of Distributed Systems syllabi from top Computer Science programs. In \"A New Course on Systems Benchmarking - For Scientists and Engineers\", Samuel Kounev - University of Würzburg, presents an overview of a new course focussed on systems benchmarking. The talk on \"Performance Engineering and Database Development at MongoDB\" by David Daly, MongoDB, presents Performance and the related properties of stability and resilience in MongoDB. In the talk \"Software Performance Engineering Education: What Topics Should be Covered?\" by Connie U. Smith, Performance Engineering Service, the author presents her vast experience with Software Perfor","PeriodicalId":433195,"journal":{"name":"Proceedings of the ACM/SPEC International Conference on Performance Engineering","volume":"61 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124651875","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"PieSlicer: Dynamically Improving Response Time for Cloud-based CNN Inference","authors":"Samuel S. Ogden, Xiangnan Kong, Tian Guo","doi":"10.1145/3427921.3450256","DOIUrl":"https://doi.org/10.1145/3427921.3450256","url":null,"abstract":"Executing deep-learning inference on cloud servers enables the usage of high complexity models for mobile devices with limited resources. However, pre-execution time-the time it takes to prepare and transfer data to the cloud-is variable and can take orders of magnitude longer to complete than inference execution itself. This pre-execution time can be reduced by dynamically deciding the order of two essential steps, preprocessing and data transfer, to better take advantage of on-device resources and network conditions. In this work, we present PieSlicer, a system for making dynamic preprocessing decisions to improve cloud inference performance using linear regression models. PieSlicer then leverages these models to select the appropriate preprocessing location. We show that for image classification applications PieSlicer reduces median and 99th percentile pre-execution time by up to 50.2ms and 217.2ms respectively when compared to static preprocessing methods.","PeriodicalId":433195,"journal":{"name":"Proceedings of the ACM/SPEC International Conference on Performance Engineering","volume":"57 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125136963","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Towards a Group Encryption Scheme Benchmark: A View on Centralized Schemes with Focus on IoT","authors":"Thomas Prantl, Peter Ten, Lukas Iffländer, Stefan Herrnleben, A. Dmitrienko, Samuel Kounev, Christian Krupitzer","doi":"10.1145/3427921.3450252","DOIUrl":"https://doi.org/10.1145/3427921.3450252","url":null,"abstract":"The number of devices connected to the Internet of Things (IoT) is continuously increasing to several billion nowadays. As those devices often share sensitive data, encryption of those data is an important issue. The pure volume of data and the complexity of communication patterns increases, and, accordingly, the importanceof group encryption is recently gaining more importance. Still, the choice of the best-suited group encryption scheme for a specific application is complicated. Benchmarks can support this choice. However, while literature distinguishes three categories for theschemes (central, decentral, and hybrid), a one-fits-all benchmark seems challenging to achieve. In this paper, we go the first step towards a structured benchmark for group encryption schemes by presenting a benchmark for centralized group encryption schemes in an IoT scenario. To this end, our benchmark includes a descriptionof workloads, a baseline scheme, a measurement setup, and metrics while also considering the requirements and features of centralized group encryption schemes.","PeriodicalId":433195,"journal":{"name":"Proceedings of the ACM/SPEC International Conference on Performance Engineering","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125463639","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}