BRISE: Energy-Efficient Benchmark Reduction

A considerable portion of research activities in computer science heavily relies on the process of benchmarking, e.g., to evaluate a hypothesis in an empirical study. The goal is to reveal how a set of independent variables (factors) influences one or more dependent variables. With a vast number of factors or a high amount of factors' values (levels), this process becomes time- and energy-consuming. Current approaches to lower the benchmarking effort suffer from two deficiencies: (1) they focus on reducing the number of factors and, hence, are inapplicable to experiments with only two factors, but a vast number of levels and (2) being adopted from, e.g., combinatorial optimization they are designed for a different search space structure and, thus, can be very wasteful. This paper provides an approach for benchmark reduction, based on adaptive instance selection and multiple linear regression. We evaluate our approach using four empirical studies, which investigate the effect made by dynamic voltage and frequency scaling in combination with dynamic concurrency throttling on the energy consumption of a computing system (parallel compression, sorting, and encryption algorithms as well as database query processing). Our findings show the effectiveness of the approach. We can save 78% of benchmarking effort, while the result's quality decreases only by 3 pp, due to using only a near-optimal configuration.