Why does the experiment fail with students? An Experience

Experimental studies in the Software Engineering (SE) domain play a vital role in identifying shreds of evidence about the effectiveness of the technique in a specific context. Successful execution of such experiments with all threats to validity considerations requires a lot of effort, planning, preparation, and experience. The selection, motivation, and experience of the subjects (specifically with the students) are considered a major potential threat. In this paper, we report various challenges, recommendations, and experiences gained while conducting an experimental study with students as subjects. The experiment was conducted to assess the effectiveness of tool support developed to evaluate the quality of requirements specification. Our results have revealed several outliers in the data and suggested to go back and see what went wrong in the design and execution process.