Evaluation Framework for Cold-Start Techniques in Large-Scale Production Settings

In recommender systems, cold-start issues are situations where no previous events (e.g., ratings), are known for certain users or items. Mitigating cold-start situations is a fundamental problem in almost any recommender system [3, 5]. In real-life, large-scale production systems, the challenge of optimizing the cold-start strategy is even greater. We present an end-to-end framework for evaluating and comparing different cold-start strategies. By applying this framework in Outbrain’s recommender system, we were able to reduce our cold-start costs by half, while supporting both offline and online settings. Our framework solves the pain of benchmarking numerous cold-start techniques using surrogate accuracy metrics on offline datasets - coupled with an extensive, cost-controlled online A/B test. In this abstract, We’ll start with a short introduction to the cold-start challenge in recommender systems. Next, we will explain the motivation for a framework for cold-start techniques. Lastly, we will then describe - step by step - how we used the framework to reduce our exploration by more than 50%.