Towards a Visualization-Driven Approach to Database Benchmarking Analysis

Abstract

Employing TPC-defined benchmarks and their derivatives is an established approach adopted by organizations to evaluate and demonstrate performance of their database management systems with the goal of increasing sales and establishing competitiveness of their products. One common challenge in the benchmarking process is the data analysis that involves large, performance datasets for characterizing a database system over underlying system configuration. In this paper, we address two different scenarios that demand detailed data analysis and are commonly found in database benchmarking process - analyzing query execution behavior when multiple streams of queries are run concurrently (typically referred as throughput phase in TPC benchmarks), and visualizing query performance with respect to different resources - cores, memory, storage. We highlight the challenges that exist in the raw data analysis space for each of these use-cases and then demonstrate how the data visualizations we have developed using Python enable insights in an easy-to-use, intuitive manner. Given that the two scenarios we cover are common across multiple benchmarks such as TPC-H, TPC-DS, TPCxBB, and their derivatives, our proposed visualizations can be adapted and used as a resource by the database benchmarking community.