{"title":"Unifying Explicit and Implicit Feedback for Rating Prediction and Ranking Recommendation Tasks","authors":"A. H. Jadidinejad, C. Macdonald, I. Ounis","doi":"10.1145/3341981.3344225","DOIUrl":"https://doi.org/10.1145/3341981.3344225","url":null,"abstract":"The two main tasks addressed by collaborative filtering approaches are rating prediction and ranking. Rating prediction models leverage explicit feedback (e.g. ratings), and aim to estimate the rating a user would assign to an unseen item. In contrast, ranking models leverage implicit feedback (e.g. clicks) in order to provide the user with a personalized ranked list of recommended items. Several previous approaches have been proposed that learn from both explicit and implicit feedback to optimize the task of ranking or rating prediction at the level of recommendation algorithm. Yet we argue that these two tasks are not completely separate, but are part of a unified process: a user first interacts with a set of items and then might decide to provide explicit feedback on a subset of items. We propose to bridge the gap between the tasks of rating prediction and ranking through the use of a novel weak supervision approach that unifies both explicit and implicit feedback datasets. The key aspects of the proposed model is that (1) it is applied at the level of data pre-processing and (2) it increases the representation of less popular items in recommendations while maintaining reasonable recommendation performance. Our experimental results - on six datasets covering different types of heterogeneous user's interactions and using a wide range of evaluation metrics - show that, our proposed approach can effectively combine explicit and implicit feedback and improve the effectiveness of the baseline explicit model on the ranking task by covering a broader range of long-tail items.","PeriodicalId":173154,"journal":{"name":"Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval","volume":"9 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126479659","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Generalising Kendall's Tau for Noisy and Incomplete Preference Judgements","authors":"Riku Togashi, T. Sakai","doi":"10.1145/3341981.3344246","DOIUrl":"https://doi.org/10.1145/3341981.3344246","url":null,"abstract":"We propose a new ranking evaluation measure for situations where multiple preference judgements are given for each item pair but they may be noisy (i.e., some judgements are unreliable) and/or incomplete (i.e., some judgements are missing). While it is generally easier for assessors to conduct preference judgements than absolute judgements, it is often not practical to obtain preference judgements for all combinations of documents. However, this problem can be overcome if we can effectively utilise noisy and incomplete preference judgements such as those that can be obtained from crowdsourcing. Our measure, η, is based on a simple probabilistic user model of the labellers which assumes that each document is associated with a graded relevance score for a given query. We also consider situations where multiple preference probabilities, rather than preference labels, are given for each document pair. For example, in the absence of manual preference judgements, one might want to employ an ensemble of machine learning techniques to obtain such estimated probabilities. For this scenario, we propose another ranking evaluation measure called η_p $. Through simulated experiments, we demonstrate that our proposed measures η and η_p$ can evaluate rankings more reliably than τmbox- b$, a popular rank correlation measure.","PeriodicalId":173154,"journal":{"name":"Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval","volume":"116 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129910559","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Behrooz Mansouri, Shaurya Rohatgi, Douglas W. Oard, Jian Wu, C. Lee Giles, R. Zanibbi
{"title":"Tangent-CFT: An Embedding Model for Mathematical Formulas","authors":"Behrooz Mansouri, Shaurya Rohatgi, Douglas W. Oard, Jian Wu, C. Lee Giles, R. Zanibbi","doi":"10.1145/3341981.3344235","DOIUrl":"https://doi.org/10.1145/3341981.3344235","url":null,"abstract":"When searching for mathematical content, accurate measures of formula similarity can help with tasks such as document ranking, query recommendation, and result set clustering. While there have been many attempts at embedding words and graphs, formula embedding is in its early stages. We introduce a new formula embedding model that we use with two hierarchical representations, (1) Symbol Layout Trees (SLTs) for appearance, and (2) Operator Trees (OPTs) for mathematical content. Following the approach of graph embeddings such as DeepWalk, we generate tuples representing paths between pairs of symbols depth-first, embed tuples using the fastText n-gram embedding model, and then represent an SLT or OPT by its average tuple embedding vector. We then combine SLT and OPT embeddings, leading to state-of-the-art results for the NTCIR-12 formula retrieval task. Our fine-grained holistic vector representations allow us to retrieve many more partially similar formulas than methods using structural matching in trees. Combining our embedding model with structural matching in the Approach0 formula search engine produces state-of-the-art results for both fully and partially relevant results on the NTCIR-12 benchmark. Source code for our system is publicly available.","PeriodicalId":173154,"journal":{"name":"Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval","volume":"9 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114292582","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Term Discrimination Value for Cross-Language Information Retrieval","authors":"Ali Montazeralghaem, Razieh Rahimi, J. Allan","doi":"10.1145/3341981.3344252","DOIUrl":"https://doi.org/10.1145/3341981.3344252","url":null,"abstract":"Term discrimination value is among the three basic heuristics exploited, directly or indirectly, in almost all ranking models for ad-hoc Information Retrieval (IR). Query term discrimination in monolingual IR is usually estimated based on document or collection frequency of terms. In the query translation approach for CLIR, the discrimination value of a query term needs to be estimated based on document or collection frequencies of its translations, which is more challenging. We show that the existing estimation models do not correctly estimate and adequately reflect the difference between the discrimination power of query terms, which hurts retrieval performance. We then propose a new model to estimate discrimination values of query terms for CLIR and empirically demonstrate its impact in improving the CLIR performance.","PeriodicalId":173154,"journal":{"name":"Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval","volume":"16 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121907583","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Alberto Purpura, Marco Maggipinto, G. Silvello, Gian Antonio Susto
{"title":"Probabilistic Word Embeddings in Neural IR: A Promising Model That Does Not Work as Expected (For Now)","authors":"Alberto Purpura, Marco Maggipinto, G. Silvello, Gian Antonio Susto","doi":"10.1145/3341981.3344217","DOIUrl":"https://doi.org/10.1145/3341981.3344217","url":null,"abstract":"In this paper, we discuss how a promising word vector representation based on Probabilistic Word Embeddings (PWE) can be applied to Neural Information Retrieval (NeuIR). We illustrate PWE pros for text retrieval, and identify the core issues which prevent a full exploitation of their potential. In particular, we focus on the application of elliptical probabilistic embeddings, a type of PWE, to a NeuIR system (i.e., MatchPyramid). The main contributions of this paper are: (i) an analysis of the pros and cons of PWE in NeuIR; (ii) an in-depth comparison of PWE against pre-trained Word2Vec, FastText and WordNet word embeddings; (iii) an extension of the MatchPyramid model to take advantage of broader word relations information from WordNet; (iv) a topic-level evaluation of the MatchPyramid ranking models employing the considered word embeddings. Finally, we discuss some lessons learned and outline some open research problems to employ PWE in NeuIR systems more effectively.","PeriodicalId":173154,"journal":{"name":"Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval","volume":"62 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124413861","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"An Assumption-Free Approach to the Dynamic Truncation of Ranked Lists","authors":"Yen-Chieh Lien, Daniel Cohen, W. Bruce Croft","doi":"10.1145/3341981.3344234","DOIUrl":"https://doi.org/10.1145/3341981.3344234","url":null,"abstract":"In traditional retrieval environments, a ranked list of candidate documents is produced without regard to the number of documents. With the rise in interactive IR as well as professional searches such as legal retrieval, this results in a substantial ranked list which is scanned by a user until their information need is satisfied. Determining the point at which the ranking model has low confidence in the relevance score is a challenging, but potentially very useful, task. Truncation of the ranked list must balance the needs of the user with the confidence of the retrieval model. Unlike query performance prediction where the task is to estimate the performance of a model based on an initial query and a given set documents, dynamic truncation minimizes the risk of viewing a non-relevant document given an external metric by estimating the confidence of the retrieval model using a distribution over its already calculated output scores, and subsequently truncating the ranking at that position. In this paper, we propose an assumption-free approach to learning a non-parametric score distribution over any retrieval model and demonstrate the efficacy of our method on Robust04, significantly improving user defined metrics compared to previous approaches.","PeriodicalId":173154,"journal":{"name":"Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval","volume":"149 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122946286","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Neural Attentive Cross-Domain Recommendation","authors":"Dimitrios Rafailidis, F. Crestani","doi":"10.1145/3341981.3344214","DOIUrl":"https://doi.org/10.1145/3341981.3344214","url":null,"abstract":"Nowadays, users open multiple accounts on social media platforms and e-commerce sites, expressing their personal preferences on different domains. However, users' behaviors change across domains, depending on the content that users interact with, such as movies, music, clothing and retail products. The main challenge is how to capture users' complex preferences when generating cross-domain recommendations, that is exploiting users' preferences from source domains to generate recommendations in a target domain. In this study, we propose a Neural Attentive Cross-domain model, namely NAC. We design a neural architecture, to carefully transfer the knowledge of user preferences across domains by taking into account the cross-domain latent effects of multiple source domains on users' selections in a target domain. In addition, we introduce a cross-domain behavioral attention mechanism to adaptively perform the weighting of users' preferences from the source domains, and consequently generate accurate cross-domain recommendations. Our experiments on ten cross-domain recommendation tasks show that the proposed NAC model achieves higher recommendation accuracy than other state-of-the-art methods for both ordinary and cold-start users. Furthermore, we study the effect of the proposed cross-domain behavioral attention mechanism and show that it is a key factor to our model's performance.","PeriodicalId":173154,"journal":{"name":"Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval","volume":"7 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131469927","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"A Multi-Source Collection of Event-Labeled News Documents","authors":"I. Mele, F. Crestani","doi":"10.1145/3341981.3344253","DOIUrl":"https://doi.org/10.1145/3341981.3344253","url":null,"abstract":"In this paper, we present a collection of news documents labeled at the level of crisp events. Compared to other publicly-available collections, our dataset is made of heterogeneous documents published by popular news channels on different platforms in the same temporal window and, therefore, dealing with roughly the same events and topics. The collection spans 4 months and comprises 147K news documents from 27 news streams, i.e., 9 different channels and 3 platforms: Twitter, RSS portals, and news websites. We also provide relevance labels of news documents for some selected events. These relevance judgments were collected using crowdsourcing. The collection can be useful to researchers investigating challenging news-mining tasks, such as event detection and tracking, multi-stream analysis, and temporal analysis of news publishing patterns.","PeriodicalId":173154,"journal":{"name":"Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval","volume":"67 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131409364","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"What Do We Mean by Theory in Information Retrieval?","authors":"D. Kelly","doi":"10.1145/3341981.3343870","DOIUrl":"https://doi.org/10.1145/3341981.3343870","url":null,"abstract":"Theory comes in many forms. In some fields, mathematical statements are used to communicate theory, while in others, verbal statements are used. Some fields rely heavily on models, such as physical, mechanical or stochastic, while other fields rely on simulation. As an area of study, information retrieval (IR) addresses topics and problems that can be informed by a wide-variety of theories and models, including those used to describe the actions of machines, as well as those used to explain the behaviors of humans and systems. While theoretical research is often presented as being at odds with empirical research, in reality, they cannot be separated. In this talk, I will review several theories and models that have guided select IR research, and discuss the ways researchers have exercised, explored and tested theories. I will also discuss the role of theories in IR research, and present criteria that can be used to reason about whether an IR theory is useful. I will argue that we should demand more theory from IR research. In particular, while the absence of theory does not prevent us from doing research, it does restrict our findings to a narrow slice of time.","PeriodicalId":173154,"journal":{"name":"Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval","volume":"156 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123282561","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Relevance Modeling with Multiple Query Variations","authors":"Xiaolu Lu, Oren Kurland","doi":"10.1145/3341981.3344224","DOIUrl":"https://doi.org/10.1145/3341981.3344224","url":null,"abstract":"The generative theory for relevance and its operational manifestation --- the relevance model --- are based on the premise that a single query is used to represent an information need for retrieval. In this work, we extend the theory and devise novel techniques for relevance modeling using as set of query variations representing the same information need. Our new approach is based on fusion at the term level, the model level, or the document-list level. We theoretically analyze the connections between these methods and provide empirical support of their equivalence using TREC datasets. Specifically, our new approach of inducing relevance models from multiple query variations substantially outperforms relevance model induction from a single query which is the standard practice. Our approach also outperforms fusion over multiple query variations, which is currently one of the best known baselines for several commonly used test collections.","PeriodicalId":173154,"journal":{"name":"Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval","volume":"32 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124457825","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}