Can results from software engineering experiments be safely combined?

Abstract

Deriving reliable empirical results from a single experiment is an unlikely event. Hence to progress, multiple experiments must be undertaken per hypothesis and the subsequent results effectively combined to produce a single reliable conclusion. Other disciplines use meta-analytic techniques to achieve this result. The treatise of the paper is: can meta-analysis be successfully applied to current software engineering experiments? The question is investigated by examining a series of experiments, which themselves investigate: which defect detection technique is best? Applying meta-analysis techniques to the software engineering data is relatively straightforward, but unfortunately the results are highly unstable, as the meta-analysis shows that the results are highly disparate and don't lead to a single reliable conclusion. The reason for this deficiency is the excessive variation within various components of the experiments. Finally, the paper describes a number of recommendations for controlling and reporting empirical work to advance the discipline towards a position where meta-analysis can be profitably employed.