On the State of NoSQL Benchmarks

The proliferation of Big Data systems and namely NoSQL databases has resulted in a tremendous heterogeneity in its offerings. It has become increasingly difficult to compare and select the most optimal NoSQL storage technology. Current benchmark efforts, such as the Yahoo! Cloud Serving Benchmark (YCSB), evaluate simple read and write operations on a primary key. However, while YCSB has become the de-facto benchmark solution for practitioners and NoSQL vendors, it is lacking in capabilities to extensively evaluate specific NoSQL solutions. In this paper, we present a systematic survey of current NoSQL benchmarks, in which we identify a clear gap in benchmarking more advanced workloads (e.g. nested document search) for features specific to NoSQL database families (e.g. document stores). Secondly, based on our survey, we discuss the strengths and weaknesses of the different benchmark design approaches, and argue in favor of a benchmark suite that targets specific families of NoSQL databases yet still allows overall comparison of databases in terms of their commonalities.