Perspector: Benchmarking Benchmark Suites

Abstract

Estimating the quality of a benchmark suite is a non-trivial task. A poorly selected or improperly configured bench-mark suite can present a distorted picture of the performance of the evaluated framework. With computing venturing into new domains, the total number of benchmark suites available is increasing by the day. Researchers must evaluate these suites quickly and decisively for their effectiveness. We present Perspector, a novel tool to quantify the performance of a benchmark suite. Perspector comprises novel metrics to characterize the quality of a benchmark suite. It provides a math-ematical framework for capturing some qualitative suggestions and observations made in prior work. The metrics are generic and domain-agnostic. Furthermore, our tool can be used to compare the efficacy of one suite vis-a-vis other benchmark suites, systematically and rigorously create a suite of workloads, and appropriately tune them for a target system.