A primer on power and sample size calculations for randomisation inference with experimental data

Abstract

This paper revisits the problem of power analysis and sample size calculations in randomised experiments, with a focus on settings where inference on average treatment effects is conducted using randomisation tests. While standard formulas based on the two-sample $t$-test are widely used in practice, we show that these calculations may yield misleading results when directly applied to randomisation-based inference – unless certain assumptions are met. We demonstrate that differences in potential outcome variances or unequal group sizes can distort the behaviour of the randomisation test, leading to incorrect power and flawed sample size calculations. However, a simple adjustment – studentising the test statistic – restores the validity of the randomisation test in large samples. This adjustment allows researchers to safely apply standard power and sample size formulas, even when using randomisation inference. We extend these results to a range of experimental designs commonly used in applied economics, including stratified randomisation, matched pairs and cluster-randomised trials. Throughout, we provide practical guidance to help researchers ensure that their design-stage calculations remain valid under the inferential methods they plan to use.