MalikStone — A context-aware light-weight benchmark for P2P computing

Abstract

Benchmarking has been used by performance engineers for over three decades to gain better insight into system performance. Numerous benchmarks are used in industry to characterize the performance of standalone systems on the one hand, to measure the computational efficiency of massively parallel systems such as clusters, grids and supercomputers, on the other hand. One of the common features of performance evaluation of small or large standalone systems is that performance engineers have direct and full control over these systems. The peer-to-peer (P2P) paradigm, which involves volunteer participation for execution of lengthy and complex scientific computations, operates in an uncontrolled environment. The performance evaluation of participating nodes operating in an uncontrolled P2P environment is a challenging task. Currently, two traditional synthetic benchmarks (Dhrystone and Whetstone) are used as part of Berkeley Open Infrastructure for Network Computing (BOINC)'s Accounting System for granting credits to the participating nodes. The analysis of the performance data obtained from running Dhrystone and Whetstone on general purpose computers has highlighted the limitations of these benchmarks. This study proposes a new synthetic light-weight benchmark - MalikStone, which is representative of large P2P projects and has been specifically designed in view the dynamic nature and challenges of the P2P paradigm. The benchmark integrates the strengths of existing synthetic benchmarks and provides more detailed insight into system performance by capturing the context in which the benchmarking was performed. The results of this newly designed benchmark have been found to be encouraging. Compared with existing synthetic benchmarks, Dhrystone and Whetstone, and the SPEC CPU2006 benchmarking suite, we highlight MalikStone's superiority in characterizing system performance in P2P settings.