Irreproducible benchmarks might be sometimes helpful

Abstract

Historically, benchmarks have been used for commercial purposes. A customer develops or selects a benchmark that generates a load that is considered to be typical for her/his applications. The benchmark is executed on various machines to find the most appropriate vendor and machine. In OS research, we use benchmarks for completely different purposes that can be subsumed under the headline “understanding a complex system”: