On the need to define community agreements for controlled experiments with human subjects: a discussion paper

Abstract

While it looks like controlled trials with human involvement are increasingly applied in software science, there are few explicitly documented community standards in regard to their design or approach. This leads to a number of problems: experimenters cannot be sure whether an experiment they perform does represent the current state-of-the-art, reviewers have no guidelines to check whether a critique they have in mind is valid or not, and readers from experiments have hardly any chance to check whether the results of an experiment they are reading should be taken seriously. This paper discusses the problem of missing community standards for empirical studies in computer science and makes a first proposal with respect to subjects, training, measurements, experimental designs, and documentation. The overall goal of this paper is to begin a discussion on this issue.