How Common are Controlled Experiments with Student Participants in Requirements Engineering?: A Systematic Mapping Study on the Use and Reporting of Graduate and Undergraduate Students in Requirements Engineering Experiments

Abstract

[Context] In requirements engineering research, emphasis is given to sound evaluations of new approaches. While industry surveys or industrial case studies are preferred to evaluate industrial applicability, controlled experiments with student participants are commonly used to determine measurements such as effectiveness and efficiency of a proposed approach. [Objectives] In this paper, we elaborate on the current state of the art of controlled experiments using student participants. As threats regarding the generalizability are quite obvious, we want to determine how widespread controlled experiments with student participants are and in which settings they are used. [Methods] This paper reports on a systematic mapping study using requirements engineering specific conferences and a journal as data sources. We scanned requirements engineering papers published in the years 2010-2015 and investigated all papers reporting student experiments in detail. [Results] From 444 papers under investigation 31 report results from controlled experiments. Regarding threats to validity it can be observed that most papers report on threats to validity and commonly different categories of threats to validity are thoroughly discussed. However, many experiments lack information regarding the students' recruitment and other important factors. [Conclusions] Student participation in requirements engineering experiments can be seen as a common evaluation approach and common threats to validity are appropriately discussed. In contrast, student specific threats to validity in experiments with student participants, such as the recruitment strategy and the bonuses given, are commonly not reported.