Multiprogramming system performance measurement and analysis

Abstract

Why " . . . design without evaluation usually is inadequate." "Simulation... is applicable wherever we have a certain degree of understanding of the process to be simulated." "The key to performance evaluation as well as to systems design is an understanding of what systems are and how they work." "The purpose of measurment is insight, not numbers." Why should we spend time and money analyzing the performance of computer systems or computer programs? These systems or programs have been debugged. They work. They were designed for optimum performance by competent people who are just as convinced that their performance is optimum as they are that the program or system is logically correct. Why then should we analyze performance? There are three main reasons: 1. There may be performance bugs in a program. Performance bugs are the result of errors in evaluation or judgment on performance optimization. We have no reason to suspect that performance bugs are any less frequent or less serious than logical bugs. Thus, if the performance of a program or a system is important then it should be performance debugged by measurement and analysis. 2. If a new or better system or program is to be designed, then a good, quantitative understanding of the performance of previous systems is necessary to avoid performance bugs in the new design. 3. If an important program or system is intolerably slow, then the real reasons for its poor performance must be found by measurement and analysis. Otherwise time and money may be spent correcting many obvious but minor inefficiencies with no great effect on overall performance. Worse yet, the whole thing may be reimplemented with all key bugs preserved! In all three of these reasons the objective of performance analysis is to understand the unknown. We're looking for performance bugs. If we knew what these bugs were and what they cost in performance, then we wouldn't have to look for them. But because we are looking for unknown performance limiters, we don't know in advance what performance gains can be made by finding and fixing these bugs. We don't even know how hard or how easy it will be to fix these bugs after we find them. Thus, there is no way of predicting the performance payoff from the time and money spent in performance analysis. This time and money must be spent at risk, essentially on the basis of faith that the payoffs from performance analysis will exceed the value of any alternative way of spending this time and money. This is nothing new. This "faith" concept is well understood and accepted in the scientific and engineering communities. One of the purposes of this paper is to demonstrate that analysis pays off in programming to at least the same degree as in engineering and science.