More transparency in testing results: Towards an open collective knowledge base

Abstract

We are currently witnessing an explosion of advances in database technology, that cover all phases of database application design: non-functional requirements, conceptual modeling, logical modeling, deployment, physical design and exploitation. Researchers and engineers cooperate to integrate these advances in the database design. Their proposed solutions have to be confronted with similar studies whose results have been either published in scientific papers or on specific websites such as that of TPC (the Transaction Processing Council). Recently, several researchers have highlighted difficulties in reproducing the results of existing studies. As a consequence, certain research communities require that the environment and the results of the testing activities have to be published in order to facilitate their reproduction, to publish their simulator environments, in order to allow evaluators to perform by their own the experiments. This is quite similar to the Volkswagen fake pollution controls scandal. In this paper, we firstly, advocate the transparency of testing in the database field. Secondly, we propose the use of a repository dedicated to store testing environment and results. The environment includes used data sets, deployment platform, non-functional requirements, used algorithms, hypotheses, etc. and the results completed with their measurement units.