Statistical Challenges for Studying Replication

Abstract

Recent empirical research has questioned the replicability of scientific findings in various fields, including medicine, economics, and psychology. This research has also revealed that there is no clear-cut definition or standard analysis methods for replication. As a result, there has been substantial ambiguity over the proper way to design and analyze replication studies. This talk describes statistical considerations for studying replication, and examines their implications. It identifies some surprising statistical strengths and limitations of previous research, including the use of statistical methods with surprisingly high error rates. It then argues that such issues can be avoided in future efforts by taking into account key statistical considerations in the planning and analysis of replication studies.