Enhancing Counterfactual Evaluation and Learning for Recommendation Systems

Evaluating recommendation systems is a task of utmost importance and a very active research field. While online evaluation is the most reliable evaluation procedure, it may also be too expensive to perform, if not unfeasible. Therefore, researchers and practitioners resort to offline evaluation. Offline evaluation is much more efficient and scalable, but traditional approaches suffer from high bias. This issue led to the increased popularity of counterfactual techniques. These techniques are used for evaluation and learning in recommender systems and reduce the bias in offline evaluation. While counterfactual approaches have a solid statistical basis, their application to recommendation systems is still in a preliminary research phase. In this paper, we identify some limitations of counterfactual techniques applied to recommender systems, and we propose possible ways to overcome them.