Cost-Aware Stage-Based Experimentation: Challenges and Emerging Results

Abstract

Experimentation at post-deployment phases (in production environments) can be a powerful tool for both learning how a deployed system operates and how it is being used. Though this knowledge is invaluable for optimization of the system, collecting it may require long time and experiments may even worsen the system with negative effects on users and business. This calls for methods for performing experimentation in production environments that balance the profit of experimentation with its cost. In this paper, we describe related challenges and our emerging results towards cost-aware stage-based experimentation. In particular, we aim for performing experiments that optimize towards their profit while making sure that the overall experimentation cost (e.g. total experimentation time) stays within given bounds. First, we illustrate the challenges and needs of such experimentation in two use cases from different domains. Second, we describe the main concepts behind our method in a semi-formal notation. Third, we exemplify the method by applying it in the two use cases and we report interesting first results.