Estimation Errors in Network A/B Testing Due to Sample Variance and Model Misspecification

Abstract

Companies that offer services on the Web often rely on randomized experiments known as A/B tests for assessing the impact of development and business decisions. During an experiment, each user is randomly redirected to one of two versions of the website, called treatments. Several response models were proposed to describe the behavior of a user in a social network website as a function of the treatment assigned to her and to her neighbors. However, there is no consensus as to which model should be applied to a given dataset. In this work, we propose a new response model, derive theoretical limits for the estimation error of several models, and obtain empirical results for cases where the response model was misspecified.