Benchmarking interactive systems: Calibrating the model

A methodology for benchmarking dedicated, interactive systems has been developed at The MITRE Corporation. This methodology uses a synthetic program model of the application which runs on the proposed hardware/operating system configurations and is driven by a statistically derived load. System performance is measured by analyzing the synthetic transaction response times. The methodology yields assurances to a buyer that the benchmarked system has at least an a priori defined amount of computer power available for applications-oriented software. This paper examines the methodology and the problems that were encountered and solutions which have been used in calibrating a benchmark model for a specific application. The benchmark was designed to model a large interactive information processing application on a procurement requiring loosely-coupled (no shared memory) multicomputer systems. The model consists of a set of interacting synthetic program cells, each composed of several abstractly defined components. The model is maintained in a very high level language that is automatically translated into a standard High Order Language (typically FORTRAN or COBOL) for delivery to the competing vendors. These delivered model cells contain automatically generated size and time filler code that “calibrate” the cells to consume the appropriate CPU time and memory space as defined by the abstract size units after accounting for each vendor's hardware and proposed system design.