Variance-Weighted Estimators to Improve Sensitivity in Online Experiments

Abstract

As companies increasingly rely on experiments to make product decisions, precisely measuring changes in key metrics is important. Various methods to increase sensitivity in experiments have been proposed, including methods that use pre-experiment data, machine learning, and more advanced experimental designs. However, prior work has not explored modeling heterogeneity in the variance of individual experimental users. We propose a more sensitive treatment effect estimator that relies on estimating the individual variances of experimental users using pre-experiment data. We show that that weighted estimators using individual-level variance estimates can reduce the variance of treatment effect estimates, and prove that the coefficient of variation of the sample population variance is a sufficient statistic for determining the scale of possible variance reduction. We provide empirical results from case studies at Facebook demonstrating the effectiveness of this approach, where the average experiment achieved a 17% reduction in variance with minimal impact on bias.