Keynote 2: Musings on the Holy Grail of Reproducibility

Abstract

Disciplines as diverse as psychology, physics, marketing, and medicine have, for the past few years, been going through a soul-searching over the “reproducibility crisis”. According to a recent survey in Nature, over 70% of researchers have failed in reproducing another scientist’s results and more than half have failed in trying to reproduce their own results. But replication of scientific results is the heart of the scientific method; without this cornerstone we do not have science, we have faith and mysticism. Note, however, that reproducibility comes at a steep cost: more rigor, more scrutiny, and tightened controls on what is considered a publishable result will doubtless burden scientists and slow the pace of innovation. In this talk I will discuss the roots of replication problems-replication bias, null aversion, and incentive structures for researchers-and their implications on reproducibility for the field of software engineering. Finally, I will present a few ideas on how we can think about improving the state of our discipline.