Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval最新文献_第2页

Offline Evaluation without Gain 无增益离线评估

Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval Pub Date : 2020-09-14 DOI: 10.1145/3409256.3409816

C. Clarke, Alexandra Vtyurina, Mark D. Smucker

{"title":"Offline Evaluation without Gain","authors":"C. Clarke, Alexandra Vtyurina, Mark D. Smucker","doi":"10.1145/3409256.3409816","DOIUrl":"https://doi.org/10.1145/3409256.3409816","url":null,"abstract":"We propose a simple and flexible framework for offline evaluation based on a weak ordering of results (which we call \"partial preferences\") that define a set of ideal rankings for a query. These partial preferences can be derived from from side-by-side preference judgments, from graded judgments, from a combination of the two, or through other methods. We then measure the performance of a ranker by computing the maximum similarity between the actual ranking it generates for the query and elements of this ideal result set. We call this measure the \"compatibility\" of the actual ranking with the ideal result set. We demonstrate that compatibility can replace and extend current offline evaluation measures that depend on fixed relevance grades that must be mapped to gain values, such as NDCG. We examine a specific instance of compatibility based on rank biased overlap (RBO). We experimentally validate compatibility over multiple collections with different types of partial preferences, including very fine-grained preferences and partial preferences focused on the top ranks. As well as providing additional insights and flexibility, compatibility avoids shortcomings of both full preference judgments and traditional graded judgments.","PeriodicalId":430907,"journal":{"name":"Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval","volume":"14 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-09-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133288925","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 18

Analyzing the Influence of Bigrams on Retrieval Bias and Effectiveness 双图对检索偏差和有效性的影响分析

Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval Pub Date : 2020-09-14 DOI: 10.1145/3409256.3409831

Abdulaziz Alqatan, L. Azzopardi, Yashar Moshfeghi

{"title":"Analyzing the Influence of Bigrams on Retrieval Bias and Effectiveness","authors":"Abdulaziz Alqatan, L. Azzopardi, Yashar Moshfeghi","doi":"10.1145/3409256.3409831","DOIUrl":"https://doi.org/10.1145/3409256.3409831","url":null,"abstract":"Prior work on using retrievability measures in the evaluation of information retrieval (IR) systems has laid out the foundations for investigating the relationship between retrieval effectiveness and retrieval bias. While various factors influencing bias have been examined, there has been no work examining the impact of using bigram within the index on retrieval bias. Intuitively, how the documents are represented, and what terms they contain, will influence whether they are retrievable or not. In this paper, we investigate how the bias of a system changes depending on how the documents are represented using unigrams, bigrams or both. Our analysis of three different retrieval models on three TREC collections, shows that using a bigram only representation results in the lowest bias compared to unigram only representation, but at the expense of retrieval effectiveness. However, when both representations are combined it results in reducing the overall bias, as well as increasing effectiveness. These findings suggest that when configuring and indexing the collection, that the bag-of-words approach (unigrams), should be augmented with bigrams to create better and fairer retrieval systems.","PeriodicalId":430907,"journal":{"name":"Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval","volume":"52 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-09-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132781852","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 3

Training Data Optimization for Pairwise Learning to Rank 成对学习排序的训练数据优化

Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval Pub Date : 2020-09-14 DOI: 10.1145/3409256.3409824

Hojae Han, Seung-won Hwang, Young-In Song, Siyeon Kim

{"title":"Training Data Optimization for Pairwise Learning to Rank","authors":"Hojae Han, Seung-won Hwang, Young-In Song, Siyeon Kim","doi":"10.1145/3409256.3409824","DOIUrl":"https://doi.org/10.1145/3409256.3409824","url":null,"abstract":"This paper studies data optimization for Learning to Rank (LtR), by dropping training labels to increase ranking accuracy. Our work is inspired by data dropout, showing some training data do not positively influence learning and are better dropped out, despite a common belief that a larger training dataset is beneficial. Our main contribution is to extend this intuition for noisy- and semi- supervised LtR scenarios: some human annotations can be noisy or out-of-date, and so are machine-generated pseudo-labels in semi- supervised scenarios. Dropping out such unreliable labels would contribute to both scenarios. State-of-the-arts propose Influence Function (IF) for estimating how each training instance affects learn- ing, and we identify and overcome two challenges specific to LtR. 1) Non-convex ranking functions violate the assumptions required for the robustness of IF estimation. 2) The pairwise learning of LtR incurs quadratic estimation overhead. Our technical contributions are addressing these challenges: First, we revise estimation and data optimization to accommodate reduced reliability; Second, we devise a group-wise estimation, reducing cost yet keeping accuracy high. We validate the effectiveness of our approach in a wide range of ad-hoc information retrieval benchmarks and real-life search engine datasets in both noisy- and semi-supervised scenarios.","PeriodicalId":430907,"journal":{"name":"Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval","volume":"34 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-09-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116550629","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 3

Cluster-Based Document Retrieval with Multiple Queries 基于集群的多查询文档检索

Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval Pub Date : 2020-09-14 DOI: 10.1145/3409256.3409825

Kfir Bernstein, Fiana Raiber, Oren Kurland, J. Culpepper

{"title":"Cluster-Based Document Retrieval with Multiple Queries","authors":"Kfir Bernstein, Fiana Raiber, Oren Kurland, J. Culpepper","doi":"10.1145/3409256.3409825","DOIUrl":"https://doi.org/10.1145/3409256.3409825","url":null,"abstract":"The merits of using multiple queries representing the same information need to improve retrieval effectiveness have recently been demonstrated in several studies. In this paper we present the first study of utilizing multiple queries in cluster-based document retrieval; that is, using information induced from clusters of similar documents to rank documents. Specifically, we propose a conceptual framework of retrieval templates that can adapt cluster-based document retrieval methods, originally devised for a single query, to leverage multiple queries. The adaptations operate at the query, document list and similarity-estimate levels. Retrieval methods are instantiated from the templates by selecting, for example, the clustering algorithm and the cluster-based retrieval method. Empirical evaluation attests to the merits of the retrieval templates with respect to very strong baselines: state-of-the-art cluster-based retrieval with a single query and highly effective fusion of document lists retrieved for multiple queries. In addition, we present findings about the impact of the effectiveness of queries used to represent an information need on (i) cluster hypothesis test results, (ii) percentage of relevant documents in clusters of similar documents, and (iii) effectiveness of state-of-the-art cluster-based retrieval methods.","PeriodicalId":430907,"journal":{"name":"Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval","volume":"87 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-09-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121735033","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

Search Result Diversification with Guarantee of Topic Proportionality 搜索结果多样化与主题相称性保证

Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval Pub Date : 2020-09-14 DOI: 10.1145/3409256.3409839

Sheikh Muhammad Sarwar, Raghavendra Addanki, Ali Montazeralghaem, S. Pal, J. Allan

{"title":"Search Result Diversification with Guarantee of Topic Proportionality","authors":"Sheikh Muhammad Sarwar, Raghavendra Addanki, Ali Montazeralghaem, S. Pal, J. Allan","doi":"10.1145/3409256.3409839","DOIUrl":"https://doi.org/10.1145/3409256.3409839","url":null,"abstract":"Search result diversification based on topic proportionality considers a document as a bag of weighted topics and aims to reorder or down-sample a ranked list in a way that maintains topic proportionality. The goal is to show the topic distribution from an ambiguous query at all points in the revised list, hoping to satisfy all users in expectation. One effective approach, PM-2, greedily selects the best topic that maintains proportionality at each ranking position and then selects the document that best represents that topic. From a theoretical perspective, this approach does not provide any guarantee that topic proportionality holds in the small ranked list. Moreover, this approach does not take query-document relevance into account. We propose a Linear Programming (LP) formulation, LP-QL, that maintains topic proportionality and simultaneously maximizes relevance. We show that this approach satisfies topic proportionality constraints in expectation. Empirically, it achieves a 5.5% performance gain (significant) in terms of alpha-NDCG compared to PM-2 when we use LDA as the topic modelling approach. Furthermore, we propose LP-PM-2 that integrates the solution of LP-QL with PM-2. LP-PM-2 achieves 3.2% performance gain (significant) over PM-2 in terms of alpha-NDCG with term based topic modeling approach. All of our experiments are based on a popular web document collection, ClueWeb09 Category B, and the queries are taken from TREC Web Track's diversity task.","PeriodicalId":430907,"journal":{"name":"Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval","volume":"142 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-09-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122188372","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 4

Bias in Conversational Search: The Double-Edged Sword of the Personalized Knowledge Graph 会话搜索中的偏见:个性化知识图谱的双刃剑

Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval Pub Date : 2020-09-14 DOI: 10.1145/3409256.3409834

E. Gerritse, Faegheh Hasibi, A. D. Vries

引用次数: 21

Permutation Equivariant Document Interaction Network for Neural Learning to Rank 基于神经学习排序的排列等变文档交互网络

Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval Pub Date : 2020-09-14 DOI: 10.1145/3409256.3409819

Rama Kumar Pasumarthi, Honglei Zhuang, Xuanhui Wang, Michael Bendersky, Marc Najork

{"title":"Permutation Equivariant Document Interaction Network for Neural Learning to Rank","authors":"Rama Kumar Pasumarthi, Honglei Zhuang, Xuanhui Wang, Michael Bendersky, Marc Najork","doi":"10.1145/3409256.3409819","DOIUrl":"https://doi.org/10.1145/3409256.3409819","url":null,"abstract":"How to leverage cross-document interactions to improve ranking performance is an important topic in information retrieval research. The recent developments in deep learning show strength in modeling complex relationships across sequences and sets. It thus motivates us to study how to leverage cross-document interactions for learning-to-rank in the deep learning framework. In this paper, we formally define the permutation equivariance requirement for a scoring function that captures cross-document interactions. We then propose a self-attention based document interaction network that extends any univariate scoring function with contextual features capturing cross-document interactions. We show that it satisfies the permutation equivariance requirement, and can generate scores for document sets of varying sizes. Our proposed methods can automatically learn to capture document interactions without any auxiliary information, and can scale across large document sets. We conduct experiments on four ranking datasets: the public benchmarks WEB30K and Istella, as well as Gmail search and Google Drive Quick Access datasets. Experimental results show that our proposed methods lead to significant quality improvements over state-of-the-art neural ranking models, and are competitive with state-of-the-art gradient boosted decision tree (GBDT) based models on the WEB30K dataset.","PeriodicalId":430907,"journal":{"name":"Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval","volume":"16 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-09-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129824954","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 13

Leveraging Personalized Sentiment Lexicons for Sentiment Analysis 利用个性化情感词汇进行情感分析

Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval Pub Date : 2020-09-14 DOI: 10.1145/3409256.3409850

Dominic Seyler, Jiaming Shen, Jinfeng Xiao, Yiren Wang, Chengxiang Zhai

引用次数: 2

Utilizing Axiomatic Perturbations to Guide Neural Ranking Models 利用公理扰动指导神经排序模型

Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval Pub Date : 2020-09-14 DOI: 10.1145/3409256.3409828

Zitong Cheng, Hui Fang

{"title":"Utilizing Axiomatic Perturbations to Guide Neural Ranking Models","authors":"Zitong Cheng, Hui Fang","doi":"10.1145/3409256.3409828","DOIUrl":"https://doi.org/10.1145/3409256.3409828","url":null,"abstract":"Axiomatic approaches aim to utilize reasonable retrieval constraints to guide the search for optimal retrieval models. Existing studies have shown the effectiveness of axiomatic approaches in improving the performance through either the derivation of new basic retrieval models or modifications of existing ones. Recently, neural network models have attracted more attention in the research community. Since these models are learned from training data, it would be interesting to study how to utilize the axiomatic approaches to guide the training process so that the learned models can satisfy retrieval constraints and achieve better retrieval performance. In this paper, we propose to utilize axiomatic perturbations to construct training data sets for neural ranking models. The perturbed data sets are constructed in a way to amplify the desirable properties that any reasonable retrieval models should satisfy. As a result, the models learned from the perturbed data sets are expected to satisfy more retrieval constraints and lead to better retrieval performance. Experiment results show that the models learned from the perturbed data sets indeed perform better than those learned from the original data sets.","PeriodicalId":430907,"journal":{"name":"Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval","volume":"23 9","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-09-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132152920","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 2

Length Adaptive Regularization for Retrieval-based Chatbot Models 基于检索的聊天机器人模型的长度自适应正则化

Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval Pub Date : 2020-09-14 DOI: 10.1145/3409256.3409823

Disen Wang, Hui Fang

{"title":"Length Adaptive Regularization for Retrieval-based Chatbot Models","authors":"Disen Wang, Hui Fang","doi":"10.1145/3409256.3409823","DOIUrl":"https://doi.org/10.1145/3409256.3409823","url":null,"abstract":"Chatbots aim to mimic real conversations between humans. They have started playing an increasingly important role in our daily life. Given past conversations, a retrieval-based chatbot model selects the most appropriate response from a pool of candidates. Intuitively, based on the nature of the conversations, some responses are expected to be long and informative while others need to be more concise. Unfortunately, none of the existing retrieval-based chatbot models have considered the effect of response length. Empirical observations suggested the existing models over-favor longer candidate responses, leading to sub-optimal performance. To overcome this limitation, we propose a length adaptive regularization method for retrieval-based chatbot models. Specifically, we first predict the desired response length based on the conversation context and then apply a regularization method based on the predicted length to adjust matching scores for candidate responses. The proposed length adaptive regularization method is general enough to be applied to all existing retrieval-based chatbot models. Experiments on two public data sets show the proposed method is effective to significantly improve retrieval performance.","PeriodicalId":430907,"journal":{"name":"Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval","volume":"47 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-09-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114632657","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 4