Offline evaluation of the serendipity in recommendation systems

Offline optimization of recommender systems is a difficult task. Popular optimization criteria such as RMSE, Recall, and NDCG do not correlate much with online performance, especially when the recommendation algorithm is largely different from the one used to generate the offline data. An exciting direction of research to mitigate this problem is to use more robust optimization criteria. Serendipity is reported to be a promising proxy. However, more variants exist, and it is unclear whether they can be used as a single criterion to optimize. This paper analyzes how serendipity relates to other optimization criteria for three different recommendation algorithms. Based on our findings, we propose to modify the way serendipity is computed. We conduct experiments using three collaborative filtering algorithms: K-Nearest Neighbors, Matrix Factorization, and Embarrassingly Shallow Autoencoder $(EASE^{R})$. We also employ and evaluate the ensemble learning approach and analyze the importance of the individual components of serendipity.