Issues with conducting controlled on-line experiments for E-Commerce

Abstract

More and more on-line experiments have been done in E-Commerce in order to understand the behavior of users or customers and then apply the data analysis technique to provide business guidance. One of the techniques is A/B testing. However, there is not clear guidance on the sample size in order for us to have valuable, trustable discovery. The purpose of this work is to find out a way to group customers in the data sample in order to achieve an optimal difference between the buckets. Based on the analysis result of real data collected during joining an industry project, we think the problem is complex and the meaningful conclusions have to be drawn with caution from business experiments such as A/B testing, due to the vast variation in the data. Moreover, if we don't allocate enough samples in the treatment group, the experiment could be inconclusive even if the testing lasts for a longer enough time, such as one month.