Under What Conditions Does Evidence Travel? Assessing Context Sensitivity and Generalizability in Cash Transfer Program Effectiveness.

Abstract

The evidence-based policy movement assumes that rigorously evaluated interventions replicate across contexts to produce similar effects. This universalist assumption may be unwarranted for complex social programs, where context shapes effectiveness. This study examines generalizability using 15 cash transfer programs evaluated across 12 Middle Eastern and North African countries through four complementary approaches: leave-one-out cross-validation, permutation tests, equivalence testing, and fragility indices. Leave-one-out analysis demonstrates robust stability, with 100% of key relationships maintaining a magnitude within 25% of full-sample estimates. The correlation between transfer size and poverty reduction (r = 0.711, p < .001) shows strong consistency, and permutation tests yield p = .001, indicating genuine signals rather than chance patterns. Equivalence testing reveals that programs with nearly identical designs can produce statistically distinguishable outcomes, with confidence intervals excluding practical equivalence in 67% of matched comparisons. Fragility indices of 2 to 3 for most relationships indicate moderate robustness. These findings suggest that some design-outcome relationships show stability while context matters substantially for program effectiveness. This study offers methodological tools for assessing evidence robustness in small-sample comparative research and argues for epistemic humility in cross-context predictions.