Introduction to the symposium on reproducibility and replicability in economics: Part I

Abstract

Reproducibility and replicability are cornerstones of scientific progress, ensuring that findings can withstand scrutiny and that results hold under varied analyses. In economics, these principles support a self-correcting system, fostering more reliable empirical research and more robust foundations for policy design. However, despite the recognition of replication's importance, many empirical studies, particularly non-experimental ones, lack rigorous replication. This symposium of Economic Inquiry aims to address this gap, presenting new research that underscores the need for reproduction and replication, particularly in non-experimental studies, and proposes innovative approaches to overcoming the practical challenges of replicability in economics. Due to the volume of high-quality submissions we received, we have divided this symposium into two parts. This first part highlights key methodological advances and remaining challenges that contribute to the broader conversation on reproducibility and replicability in economics. A forthcoming second part will continue this exploration, further showcasing reproductions and replications of seminal and well-cited articles.

Our call for papers sought empirical replications, methodological advances, and theoretical contributions that enrich our understanding of replication's role in economic research. The selected papers in this first part introduce new methodological tools and provide frameworks for conducting and evaluating the effectiveness of reproductions and replications. These contributions advance our collective understanding of what it means for economics to be a replicable and self-correcting science. We provide a short summary of each article below.

The symposium's first article, “A Framework for Evaluating Reproducibility and Replicability in Economics”, proposes a structured approach to assess these core aspects of scientific reliability in economics. By distinguishing between various types of reproducibility (computational, recreate, robustness) and replicability (direct, conceptual), and introducing clear indicators to measure each, the article offers a practical and theoretically grounded framework that addresses long-standing ambiguities in the field. This contribution is particularly significant in the context of increasing efforts to improve transparency and credibility in economics research.

The article “Replication Code Availability Over Time and Across Fields: Evidence From the German Data Archive for Business and Economic Studies” provides trends in replication code availability over time and across disciplines by examining studies that used the German Socio-Economic Panel (SOEP) data, which, while restricted, is available to researchers and has been used in over 2500 articles in economics and social sciences. By concentrating on studies with large common data, the study highlights both progress and ongoing challenges in making replication code accessible and addresses a critical aspect of reproducibility infrastructure in academic research. This focus aligns with the theme of the special issue by contributing valuable insights into how the availability of replication materials influences the broader landscape of research credibility and transparency. The findings also provide actionable recommendations for institutions and researchers, emphasizing the importance of replication practices in advancing robust scientific inquiry.

In the paper titled “Underpowered Studies and Exaggerated Effects: A Replication and Re-evaluation of the Magnitude of Anchoring Effects” the authors replicate a field experiment investigating the effects of anchoring on consumer willingness to pay. The study revealed substantially smaller anchoring effects than previously reported in the literature and the authors argue that earlier studies, often underpowered, overestimated the magnitude of anchoring effects. Their results highlight the need for higher-powered studies and challenge widespread evidence of large anchoring effects, documented in previous studies.

The next two articles provide methodological insights to facilitate reproducibility and replicability.

First, the article “Dynare Replication of ‘A Model of Secular Stagnation: Theory and Quantitative Evaluation’ by Eggertsson et al. (2019)” replicates a significant macroeconomic study using the Dynare software. They validate the findings of the original work while addressing discrepancies between the original paper's equations and its Matlab implementation. This effort not only enhances transparency but also lowers the barriers for researchers engaging with complex overlapping generation models. Moreover, the article demonstrates how Dynare can handle large-scale models with occasionally binding constraints, offering practical insights into computational techniques.

Second, the article “Reducing the Replication Time for Structural Estimations: A Successful Replication of 'An Anatomy of International Trade’ Using GPU Computing” replicates a complex structural estimation model from a landmark study using GPU computing and significantly enhances computational efficiency—reducing runtime by orders of magnitude. This advancement not only validates the original findings but also provides a practical framework for addressing computational barriers in replication efforts. The inclusion of a “wide replication,” applying the original model to Chinese firm data, further underscores the article's relevance by demonstrating the robustness of the model across contexts.

Next, the study “Researchers' Degrees of Flexibility: Revisiting COVID-19 Policy Evaluations” evaluates the impacts of mobility restrictions during the early phase of the COVID-19 pandemic. The authors offer evidence that seemingly minor methodological decisions, such as outcome variable transformations and covariate selection, can significantly affect the estimated policy effects. The study emphasizes the importance of employing more robust estimation techniques and considering a wider range of methodological choices.

Finally, the article “Is Economics Self-Correcting? Replications in the American Economic Review” concludes this symposium by providing a critical examination of whether reproduction and replication in economics achieve their intended effect. Analyzing reproductions and replications published as comments in the American Economic Review from 2010 to 2020, the study reveals their limited impact on citation trends of original papers and identifies a lack of consensus about their significance among authors. This underscores broader challenges in defining and achieving robustness and replicability in economics.

The papers in this symposium emphasize the breadth and importance of reproduction and replication across different domains within economics. Through continued attention to reproducibility and replicability, particularly in complex and high-stakes areas like policy evaluation and structural modeling, the field can move closer to a standard where empirical findings are more transparent, robust, and reliable.

These studies highlight significant progress in reproduction and replication efforts, yet systemic challenges persist. Issues such as weak incentives for replication, computational and methodological barriers, and the limited influence of reproduction and replication studies continue to hinder the field's self-correcting capacity. Overcoming these challenges require a concerted and sustained effort from the discipline.

Equally important is what these studies do not address—perhaps due to their focus on replicable research. They leave open critical questions: Is there a growing reliance on proprietary or administrative data that cannot be reproduced and replicated? Or is the discipline's increasing emphasis on replication influencing research practices in a meaningful way? Understanding these dynamics is essential for strengthening the credibility and transparency of economic research.

We hope that this symposium inspires further work in reproduction and replication and fosters a culture where reproducibility is not just encouraged but integral to the research process. We extend our gratitude to all contributors and reviewers for their dedication to this mission and look forward to the continued evolution of replication science within economics.