ACM Transactions on Recommender Systems最新文献

{"title":"Exploring the Landscape of Recommender Systems Evaluation: Practices and Perspectives","authors":"Christine Bauer, Eva Zangerle, Alan Said","doi":"10.1145/3629170","DOIUrl":"https://doi.org/10.1145/3629170","url":null,"abstract":"Recommender systems research and practice are fast-developing topics with growing adoption in a wide variety of information access scenarios. In this paper, we present an overview of research specifically focused on the emphevaluation of recommender systems. We perform a systematic literature review, in which we analyze 57 papers spanning six years (2017–2022). Focusing on the processes surrounding evaluation, we dial in on the methods applied, the datasets utilized, and the metrics used. Our study shows that the predominant experiment type in research on the evaluation of recommender systems is offline experimentation and that online evaluations are primarily used in combination with other experimentation methods, e.g., an offline experiment. Furthermore, we find that only a few datasets (MovieLens, Amazon review dataset) are widely used, while many datasets are used in only a few papers each. We observe a similar scenario when analyzing the employed performance metrics—a few metrics are widely used (precision, nDCG, and Recall), while many others are used in only a few papers. Overall, our review indicates that beyond-accuracy qualities are rarely assessed. Our analysis shows that the research community working on evaluation has focused on the development of evaluation in a rather narrow scope, with the majority of experiments focusing on a few metrics, datasets, and methods.","PeriodicalId":474833,"journal":{"name":"ACM Transactions on Recommender Systems","volume":"28 5","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136158800","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"On Item-Sampling Evaluation for Recommender System","authors":"Dong Li, Ruoming Jin, Zhenming Liu, Bin Ren, Jing Gao, Zhi Liu","doi":"10.1145/3629171","DOIUrl":"https://doi.org/10.1145/3629171","url":null,"abstract":"Personalized recommender system plays a crucial role in modern society, especially in e-commerce, news, and ads area. Correctly evaluating and comparing candidate recommendation models is as essential as constructing ones. The common offline evaluation strategy is holding out some user-interacted items from training data and evaluating the performance of recommendation models based on how many items they can retrieve. Specifically, for any hold-out item or so-called target item for a user, the recommendation models try to predict the probability that the user would interact with the item, and rank it among overall items , this is called global evaluation . Intuitively, a good recommendation model would assign high probabilities to such hold-out/target items. Based on the specific ranks, some metrics like Recall @ K and NDCG @ K can be calculated to further quantify the quality of the recommender model. Instead of ranking the target items among all items, Koren [22] first proposed to rank them among a small sampled set of items , and then quantified the performance of the models, this is called sampling evaluation . Ever since then, there has been a large amount of work adopting sampling evaluation due to its efficiency and frugality. In recent work, Rendle and Krichene [24, 32] argued that the sampling evaluation is ”inconsistent” with respect to a global evaluation in terms of offline top-k metrics. In this work, we first investigate the ”inconsistent” phenomenon by taking a glance at the connections between sampling evaluation and global evaluation. We reveal the approximately linear relationship between sampling with respect to its global counterpart in terms of the top-K Recall metric. Second, we propose a new statistical perspective of the sampling evaluation - to estimate the global rank distribution of the entire population. After the estimated rank distribution is obtained, the approximation of the global metric can be further derived. Third, we extend the work of Krichene and Rendle [24], directly optimizing the error with ground truth, providing not only a comprehensive empirical study but also a rigorous theoretical understanding of the proposed metric estimators. To address the ”blind spot” issue, where accurately estimating metrics for small top-K values in sampling evaluation is challenging, we propose a novel adaptive sampling method that generalizes the expectation-maximization (EM) algorithm to this setting. Last but not least, we also study the user sampling evaluation effect. This series of works outlines a clear roadmap for sampling evaluation and establishes a foundational theoretical framework. Extensive empirical studies validate the reliability of the sampling methods presented.","PeriodicalId":474833,"journal":{"name":"ACM Transactions on Recommender Systems","volume":"165 8","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136234554","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Generating Usage-related Questions for Preference Elicitation in Conversational Recommender Systems","authors":"Ivica Kostric, Krisztian Balog, Filip Radlinski","doi":"10.1145/3629981","DOIUrl":"https://doi.org/10.1145/3629981","url":null,"abstract":"A key distinguishing feature of conversational recommender systems over traditional recommender systems is their ability to elicit user preferences using natural language. Currently, the predominant approach to preference elicitation is to ask questions directly about items or item attributes. Users searching for recommendations may not have deep knowledge of the available options in a given domain. As such, they might not be aware of key attributes or desirable values for them. However, in many settings, talking about the planned use of items does not present any difficulties, even for those that are new to a domain. In this paper, we propose a novel approach to preference elicitation by asking implicit questions based on item usage. As one of the main contributions of this work, we develop a multi-stage data annotation protocol using crowdsourcing, to create a high-quality labeled training dataset. Another main contribution is the development of four models for the question generation task: two template-based baseline models and two neural text-to-text models. The template-based models use heuristically extracted common patterns found in the training data, while the neural models use the training data to learn to generate questions automatically. Using common metrics from machine translation for automatic evaluation, we show that our approaches are effective in generating elicitation questions, even with limited training data. We further employ human evaluation for comparing the generated questions using both pointwise and pairwise evaluation designs. We find that the human evaluation results are consistent with the automatic ones, allowing us to draw conclusions about the quality of the generated questions with certainty. Finally, we provide a detailed analysis of cases where the models show their limitations.","PeriodicalId":474833,"journal":{"name":"ACM Transactions on Recommender Systems","volume":"19 4","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136234704","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Towards a Causal Decision-Making Framework for Recommender Systems","authors":"Emanuele Cavenaghi, Alessio Zanga, Fabio Stella, Markus Zanker","doi":"10.1145/3629169","DOIUrl":"https://doi.org/10.1145/3629169","url":null,"abstract":"Abstract. Causality is gaining more and more attention in the machine learning community and consequently also in recommender systems research. The limitations of learning offline from observed data are widely recognized, however, applying debiasing strategies like Inverse Propensity Weighting does not always solve the problem of making wrong estimates. This concept paper contributes a summary of debiasing strategies in recommender systems and the design of several toy examples demonstrating the limits of these commonly applied approaches. Therefore, we propose to map the causality frameworks of potential outcomes and structural causal models onto the recommender systems domain in order to foster future research and development. For instance, applying causal discovery strategies on offline data to learn the causal graph in order to compute counterfactuals or improve debiasing strategies.","PeriodicalId":474833,"journal":{"name":"ACM Transactions on Recommender Systems","volume":"23 3","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134907685","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Constrained Off-policy Learning over Heterogeneous Information for Fairness-aware Recommendation","authors":"Xiangmeng Wang, Qian Li, Dianer Yu, Qing Li, Guandong Xu","doi":"10.1145/3629172","DOIUrl":"https://doi.org/10.1145/3629172","url":null,"abstract":"Fairness-aware recommendation eliminates discrimination issues to build trustworthy recommendation systems. Existing fairness-aware approaches ignore accounting for rich user and item attributes and thus cannot capture the impact of attributes on affecting recommendation fairness. These real-world attributes severely cause unfair recommendations by favoring items with popular attributes, leading to item exposure unfairness in recommendations. Moreover, existing approaches mostly mitigate unfairness for static recommendation models, e.g., collaborative filtering. Static models can not handle dynamic user interactions with the system that reflect users’ preferences shift through time. Thus, static models are limited in their ability to adapt to user behavior shifts to gain long-run user satisfaction. As user and item attributes are largely involved in modern recommenders and user interactions are naturally dynamic, it is essential to develop a novel method that eliminates unfairness caused by attributes meanwhile embrace the dynamic modeling of user behavior shifts. In this paper, we propose Constrained Off-policy Learning over Heterogeneous Information for Fairness-aware Recommendation (Fair-HINpolicy) , which uses recent advances in context-aware off-policy learning to produce fairness-aware recommendations with rich attributes from a Heterogeneous Information Network. In particular, we formulate the off-policy learning as a Constrained Markov Decision Process (CMDP) by dynamically constraining the fairness of item exposure at each iteration. We also design an attentive action sampling to reduce the search space for off-policy learning. Our solution adaptively receives HIN-augmented corrections for counterfactual risk minimization, and ultimately yields an effective policy that maximizes long-term user satisfaction. We extensively evaluate our method through simulations on large-scale real-world datasets, obtaining favorable results compared with state-of-the-art methods.","PeriodicalId":474833,"journal":{"name":"ACM Transactions on Recommender Systems","volume":"5 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136377288","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Comprehensive Survey on Automated Machine Learning for Recommendations","authors":"Bo Chen, Xiangyu Zhao, Yejing Wang, Wenqi Fan, Huifeng Guo, Ruiming Tang","doi":"10.1145/3630104","DOIUrl":"https://doi.org/10.1145/3630104","url":null,"abstract":"Deep recommender systems (DRS) are critical for current commercial online service providers, which address the issue of information overload by recommending items that are tailored to the user’s interests and preferences. They have unprecedented feature representations effectiveness and the capacity of modeling the non-linear relationships between users and items. Despite their advancements, DRS models, like other deep learning models, employ sophisticated neural network architectures and other vital components that are typically designed and tuned by human experts. This article will give a comprehensive summary of automated machine learning (AutoML) for developing DRS models. We first provide an overview of AutoML for DRS models and the related techniques. Then we discuss the state-of-the-art AutoML approaches that automate the feature selection, feature embeddings, feature interactions, and model training in DRS. We point out that the existing AutoML-based recommender systems are developing to a multi-component joint search with abstract search space and efficient search algorithm. Finally, we discuss appealing research directions and summarize the survey.","PeriodicalId":474833,"journal":{"name":"ACM Transactions on Recommender Systems","volume":"48 4","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134906798","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Unbiased Identification of Broadly Appealing Content Using a Pure Exploration Infinitely-Armed Bandit Strategy","authors":"Maryam Aziz, Jesse Anderton, Kevin Jamieson, Alice Wang, Hugues Bouchard, Javed Aslam","doi":"10.1145/3626324","DOIUrl":"https://doi.org/10.1145/3626324","url":null,"abstract":"Podcasting is an increasingly popular medium for entertainment and discourse around the world, with tens of thousands of new podcasts released on a monthly basis. We consider the problem of identifying from these newly-released podcasts those with the largest potential audiences so they can be considered for personalized recommendation to users. We first study and then discard a supervised approach due to the inadequacy of either content or consumption features for this task, and instead propose a novel non-contextual bandit algorithm in the fixed-budget infinitely-armed pure-exploration setting. We demonstrate that our algorithm is well-suited to the best-arm identification task for a broad class of arm reservoir distributions, out-competing a large number of state-of-the-art algorithms. We then apply the algorithm to identifying podcasts with broad appeal in a simulated study, and show that it efficiently sorts podcasts into groups by increasing appeal while avoiding the popularity bias inherent in supervised approaches. Finally, we study a setting in which users are more likely to stream more-streamed podcasts independent of their general appeal and find that our proposed algorithm is robust to this type of popularity bias.","PeriodicalId":474833,"journal":{"name":"ACM Transactions on Recommender Systems","volume":"26 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135481619","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Explainable Meta-Path Based Recommender Systems","authors":"Thanet Markchom, Huizhi Liang, James Ferryman","doi":"10.1145/3625828","DOIUrl":"https://doi.org/10.1145/3625828","url":null,"abstract":"Meta-paths have been popularly used to provide explainability in recommendations. Although long/complicated meta-paths could represent complex user-item connectivity, they are not easy to interpret. This work tackles this problem by introducing a meta-path translation task. The objective is to translate a meta-path to its comparable explainable meta-paths that perform similarly in terms of recommendation but have higher explainability compared to the given one. We propose a definition of meta-path explainability to determine comparable explainable meta-paths and a meta-path grammar that allows comparable explainable meta-paths to be formed in a similar way as sentences in human languages. Based on this grammar, we propose a meta-path translation model, a sequence-to-sequence (Seq2Seq) model to translate a long and complicated meta-path to its comparable explainable meta-paths. Two novel datasets for meta-path translation were generated based on two real-world recommendation datasets. The experiments were conducted on these generated datasets. The results show that our model outperformed state-of-the-art Seq2Seq baselines regarding meta-path translation and maintained a better trade-off between accuracy and diversity/readability in predicting comparable explainable meta-paths. These results indicate that our model can effectively generate a group of explainable meta-paths as alternative explanations for those recommendations based on any given long/complicated meta-path.","PeriodicalId":474833,"journal":{"name":"ACM Transactions on Recommender Systems","volume":"21 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-09-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135385293","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Efficient Exploration and Exploitation for Sequential Music Recommendation","authors":"Bruno Laporais Pereira, Pedro Dalla Vecchia Chaves, Rodrygo L. T. Santos","doi":"10.1145/3625827","DOIUrl":"https://doi.org/10.1145/3625827","url":null,"abstract":"Music streaming services heavily rely upon recommender systems to acquire, engage, and retain users. One notable component of these services are playlists, which can be dynamically generated in a sequential manner based on the user’s feedback during a listening session. Online learning to rank approaches have recently been shown effective at leveraging such feedback to learn users’ preferences in the space of song features. Nevertheless, these approaches can suffer from slow convergence as a result of their random exploration component and their session-agnostic exploitation component. To overcome these limitations, we propose a novel online learning to rank approach which efficiently explores the space of candidate recommendation models by restricting itself to the orthogonal complement of the subspace of previous underperforming exploration directions. Moreover, we propose a session-aware exploitation component which leverages the momentum of the current best model during updates. Our thorough evaluation using simulated listening sessions from two large Last.fm datasets demonstrates substantial improvements over state-of-the-art approaches in terms of early-stage performance, which results in an improved user experience during online learning. In addition, we demonstrate that long-term convergence can be further enhanced by adaptively relaxing exploration constraints along the way.","PeriodicalId":474833,"journal":{"name":"ACM Transactions on Recommender Systems","volume":"43 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135538984","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}