Experiences Conducting Experiments in Industry: The ESEIL FiDiPro Project

Abstract

The Experimental Software Engineering Industry Laboratory (ESEIL) project funded by the Finland Distinguished Professor Programme (FiDiPro) kicked off in January 2013. The aim of this research is to gain insight into whether experiments in the software industry can play the role of clinical trials in medicine, that is, field test laboratory findings, acting as the last link in the experimental chain. Besides this research goal, we believe that companies can benefit from the conducted experiments by applying the resulting evidence in their decision-making processes. Controlled experiments in laboratory settings are commonplace in software engineering, but experiments in industry are thin on the ground. Of the few existing cases, most are 1-1 (running one experiment at one company), just a few are n-1 (running n experiments at one company) and still fewer are 1-n (running one and the same experiment at n companies). So far we have conducted the same experiment at seven sites of six companies (four Finnish, one Estonian and one Spanish), and the results have been transferred so that these companies could use the local and global results in decision-making. This talk will discuss several striking and unexpected behaviours regarding both the developers participating in the experiments and the managers receiving and using the results of the experiments. The talk presents challenges and lessons learnt in recruiting participants for experiments, designing and running the experiment, as well as transferring the results.