DRM4Rec: A Doubly Robust Matching Approach for Recommender System Evaluation

Ranking is a core task in recommender systems, and various ranking metrics are designed to assess the quality of ordered item lists. A reliable evaluation method is crucial for identifying optimal recommendation algorithms, directly impacting downstream tasks such as click-through rate prediction and post-click conversion rate estimation. However, estimating these ranking metrics is a challenging task due to the existence of various biases in the historical data in recommender systems (RS), that is, the data used for evaluation may be significantly different from the target environment where the recommendation algorithm is planned to be deployed. Recent works propose using pseudo-labeling and reweighting for debiasing and thereby achieving unbiased evaluation. Despite being theoretically promising, the collected biased feedback makes the pseudo-labeling directly rely on extrapolation, and the propensity-based weighting method has a large variance in the presence of small propensities due to data sparsity commonly exists in real-world scenarios, both of them lack stability. In this paper, we propose a novel Doubly Robust Matching for Recommendation (DRM4Rec) method to achieve unbiased ranking metric evaluation. Compared to existing approaches, DRM4Rec reduces the unavoidable high variance due to the small propensities and also mitigates the direct harm to prediction performance from incorrect extrapolations. In addition, the proposed method has double robustness — it achieves unbiased ranking metric evaluation when either the imputed relevance or the learned propensities are accurate. We conduct extensive semi-synthetic and real-world experiments to evaluate three representative recommendation models, and the results show that DRM4Rec provides significant improvements for unbiased ranking metrics evaluations.