Measuring the sensitivity of difference-in-difference estimates to the parallel trends assumption

Abstract

Background. Difference-in-Difference makes a critical assumption that the changes in the outcomes, over the post-treatment period, are similar between the treated and control groups—the parallel trends assumption. Evaluation of this assumption is often done either by graphical examination or by statistical tests in the pre-treatment period. They result in a binary conclusion about the validity of the assumption. Purpose. This paper proposes a sensitivity analysis that quantifies the departure from parallel trends necessary to meaningfully change the estimated treatment effect. Results. Sensitivity analyses have an advantage over traditional parallel trends tests: they use all available data and thereby work even if only one pre-period is available, and they quantify the strength of unobserved confounder(s) required to change the conclusions of a study. Conclusions. We apply the sensitivity analysis metrics developed by Cinelli and Hazlett (2020) and illustrate them on two studies.