Researchers' degrees of flexibility: Revisiting COVID-19 policy evaluations

Abstract

Empirical research involves multiple, seemingly-minor choices that can substantially impact a study's findings. While acknowledged, the importance of these “degrees of flexibility” on published estimates is not well understood. We examine the considerable literature focused on the impacts of early COVID-19 policies on social distancing to assess the role of researchers' degrees of flexibility on the estimated effects of mobility-reducing policies. We find that estimates reported in previous studies are not robust to minor changes in typically-unexplored dimensions of the degree of flexibility space, and usual robustness tests systematically fail to detect these issues.