Selecting "best" MPP: do benchmarks really tell the whole story?

The proliferation of MPP's has made it very difficult for users to accurately assess which machine is "best" for a specific user application. Simply testing one or more kernels on a new machine is insufficient since slight program modifications can produce large variations in performance and thus could lead users to erroneous conclusions about the performance of the entire application. Most industrial applications are far too large to be easily or quickly benchmarked. Although there are many benchmark suites, can performance with any of those accurately predict performance for a specific industrial application? This is the dilemma potential MPP users face. The panel members have had extensive benchmarking experience. Topics to be discussed include: assessing MPP performance for scientific, engineering, and/or commercial data processing applications (including database and transactions processing); limitations of existing benchmark suites and performance metrics; guidelines to assess MPP performance.