Five Ways Not To Fool Yourself

Performance experiments are often used to show that a new system is better than an old system, and to quantify how much faster it is, or how much more efficient it is in the use of some resource. Frequently, these experiments come toward the end of a project and - at times - seem to be conducted more with the aim of selling the system rather than providing understanding of the reasons for the differences in performance or the scenarios in which similar improvements might be expected. Mistrust in published performance numbers follows from the suspicion that we measure what we have already optimized.