Making computer science results reproducible - A case study using Gradle and Docker

Abstract

This paper addresses two questions related to reproducibility within the context of research related to computer science. First, requirements on reproducibility are analyzed based on a survey addressed to researchers in the academic and private sector. The survey indicates a strong need for open but also easily accessible results, thus reproducing an experiment should not require too much effort. The results from the survey are then used to formulate general guidelines for making research results reproducible. In addition, this paper explores a number of existing software tools that could bring forward reproducibility in research results. After a general analysis of tools a further investigation is done via three case studies based on actual research projects which are used to evaluate the previously introduced tools. Results indicate that due to conflicting requirements, none of the presented solutions fulfills all intended goals perfectly. However, we present requirements and guidelines for making research reproducible. While the main focus of this paper is on reproducibility in computer science, the results of this paper are still valid for other fields using computation as a tool.