The impact of students' skills and experiences on empirical results: a controlled experiment with undergraduate and graduate students

In empirical software engineering research, graduate students are often seen as legitimate substitutes for industry professionals. It has been also argued in the literature that the generalizability of empirical results from experiments with undergraduate students as participants holds to a much lower extent. In this paper, we report on a controlled experiment conducted separately with graduate students and undergraduate students in order to gain deeper insights whether the results from experiments with graduates and undergraduates in the software engineering field are equal or significantly different with respect to the conclusions that can be drawn. During the experiment, the students apply a specific validation technique for behavioral requirements of embedded software. We observed that graduates were significantly more effective, efficient, and confident in their tasks than the undergraduates. Nevertheless, the experiment with undergraduates also shows significant results, even though with a smaller effect size.