Estimating Effects of Long-Term Treatments

Abstract

Randomized controlled trials (RCTs), also known as A/B tests, have become the gold standard for evaluating the effectiveness of product changes on digital platforms. Accurately estimating the effects of long-term treatments still remains a challenge. Product updates such as new user interfaces or recommendation algorithms are intended to persist in the system for an extended period. However, A/B testing is typically conducted for short durations, often less than two weeks, to facilitate rapid product iterations. Conducting lengthy experiments to capture the long-term impact of product changes becomes impractical due to potential negative impacts on user experiences, high opportunity costs associated with user traffic, and delays in decision-making processes.