Reading the fine print: what benchmarks don't tell you

Abstract

computer is the most general tool ever developed by mankind. Computers differ from other tools, like cars, in that they can be used in an almost limitless ways. A car has little purpose other than to efficiently move people and things short distances over smooth terrain. A graphics computer, on the other hand, might be used for a wide variety of complex applications including industrial package design, battle simulation, earth resource analysis, medical diagnosis and surgical planning, genetic research, or virtual reality games. During the early days of the auto industry, there was little need to provide information to help customers decide which car to buy. Nowadays, manufacturers provide all sorts of specifications to help people make purchasing decisions, including engine size, miles per gallon, time to accelerate from 0 to 50, trunk size, and so on. When combined with the price and the appearance of the car, these specifications and performance metrics can help consumers decide which of the many makes and models of car to buy. We are in the Model-T stage of developing specifications and performance metrics that help people make computer purchasing decisions. It is clear that the performance of a graphics computer has a large bearing on its ability to solve complex problems, but the problems themselves are of such a wide variety that it is extremely difficult to characterize the machine's performance for all the uses to which it might be put. We have some extremely crude ways to characterize the performance of the pieces that make up a graphics computer. But we have not yet succeeded in developing performance metrics that enable people to easily see whether a graphics computer will have sufficient performance for the problem set that is important to them. A great deal of energy has been invested in developing ways to quantify graphics performance. Have these efforts helped the industry or have they only served to confuse people even more? This panel takes a look at the attempts that have been made to develop useful measures of graphics performance and how those measurements may be used to compare systems from different vendors. More importantly, the panelists will describe the cave-ats and limitations of various benchmarking metrics and provide some real-world insight into how people might be able to apply published graphics benchmark results to their own needs. Background As a precursor to NCGA's Graphics Performance Characterization (GPC) Committee, the …