Proceedings of the 2015 International Conference on The Theory of Information Retrieval最新文献_第5页

Dynamic Test Collections for Retrieval Evaluation 用于检索评估的动态测试集合

Proceedings of the 2015 International Conference on The Theory of Information Retrieval Pub Date : 2015-09-27 DOI: 10.1145/2808194.2809470

Ben Carterette, Ashraf Bah Rabiou, M. Zengin

{"title":"Dynamic Test Collections for Retrieval Evaluation","authors":"Ben Carterette, Ashraf Bah Rabiou, M. Zengin","doi":"10.1145/2808194.2809470","DOIUrl":"https://doi.org/10.1145/2808194.2809470","url":null,"abstract":"Batch evaluation with test collections of documents, search topics, and relevance judgments has been the bedrock of IR evaluation since its adoption by Salton for his experiments on vector space systems. Such test collections have limitations: they contain no user interaction data; there is typically only one query per topic; they have limited size due to the cost of constructing them. In the last 15-20 years, it has become evident that having a log of user interactions and a large space of queries is invaluable for building effective retrieval systems, but such data is generally only available to search engine companies. Thus there is a gap between what academics can study using static test collections and what industrial researchers can study using dynamic user data. In this work we propose dynamic test collections to help bridge this gap. Like traditional test collections, a dynamic test collection consists of a set of topics and relevance judgments. But instead of static one-time queries, dynamic test collections generate queries in response to the system. They can generate other actions such as clicks and time spent reading documents. Like static test collections, there is no human in the loop, but since the queries are dynamic they can generate much more data for evaluation than static test collections can. And since they can simulate user interactions across a session, they can be used for evaluating retrieval systems that make use of session history or other user information to try to improve results.","PeriodicalId":440325,"journal":{"name":"Proceedings of the 2015 International Conference on The Theory of Information Retrieval","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131119827","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 33

Bayesian Inference for Information Retrieval Evaluation 信息检索评价中的贝叶斯推理

Proceedings of the 2015 International Conference on The Theory of Information Retrieval Pub Date : 2015-09-27 DOI: 10.1145/2808194.2809469

Ben Carterette

引用次数: 23

IR meets NLP: On the Semantic Similarity between Subject-Verb-Object Phrases IR与NLP的结合:主谓宾短语语义相似性研究

Proceedings of the 2015 International Conference on The Theory of Information Retrieval Pub Date : 2015-09-27 DOI: 10.1145/2808194.2809448

Dmitrijs Milajevs, M. Sadrzadeh, T. Roelleke

{"title":"IR meets NLP: On the Semantic Similarity between Subject-Verb-Object Phrases","authors":"Dmitrijs Milajevs, M. Sadrzadeh, T. Roelleke","doi":"10.1145/2808194.2809448","DOIUrl":"https://doi.org/10.1145/2808194.2809448","url":null,"abstract":"Measuring the semantic similarity between phrases and sentences is an important task in natural language processing (NLP) and information retrieval (IR). We compare the quality of the distributional semantic NLP models against phrase-based semantic IR. The evaluation is based on the correlation between human judgements and model scores on a distributional phrase similarity task. We experiment with four NLP and two IR model variants. On the NLP side, models vary over normalization schemes and composition operators. On the IR side, models vary with respect to estimation of the probability of a term being in a document, namely P(t|d) where only term co-occurrence information is used and P(t|d, sim) which incorporates term distributional similarity. A mixture of the two methods is presented and evaluated. For both methods, word meanings are derived from large corpora of data: the BNC and ukWaC. One of the main findings is that grammatical distributional models give better scores than the IR models. This suggests that an IR model enriched with distributional linguistic information performs better in the long standing problem in IR of document retrieval where there is no direct symbolic relationship between query and document concepts.","PeriodicalId":440325,"journal":{"name":"Proceedings of the 2015 International Conference on The Theory of Information Retrieval","volume":"387 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115481076","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 8

An Axiomatically Derived Measure for the Evaluation of Classification Algorithms 分类算法评价的公理化推导测度

Proceedings of the 2015 International Conference on The Theory of Information Retrieval Pub Date : 2015-09-27 DOI: 10.1145/2808194.2809449

F. Sebastiani

引用次数: 47

Implicit Preference Labels for Learning Highly Selective Personalized Rankers 学习高度选择性个性化排名的内隐偏好标签

Proceedings of the 2015 International Conference on The Theory of Information Retrieval Pub Date : 2015-09-27 DOI: 10.1145/2808194.2809464

Paul N. Bennett, Milad Shokouhi, R. Caruana

{"title":"Implicit Preference Labels for Learning Highly Selective Personalized Rankers","authors":"Paul N. Bennett, Milad Shokouhi, R. Caruana","doi":"10.1145/2808194.2809464","DOIUrl":"https://doi.org/10.1145/2808194.2809464","url":null,"abstract":"Interaction data such as clicks and dwells provide valuable signals for learning and evaluating personalized models. However, while models of personalization typically distinguish between clicked and non-clicked results, no preference distinctions within the non-clicked results are made and all are treated as equally non-relevant. In this paper, we demonstrate that failing to enforce a prior on preferences among non-clicked results leads to learning models that often personalize with no measurable gain at the risk that the personalized ranking is worse than the non-personalized ranking. To address this, we develop an implicit preference-based framework that enables learning highly selective rankers that yield large reductions in risk such as the percentage of queries personalized. We demonstrate theoretically how our framework can be derived from a small number of basic axioms that give rise to well-founded target rankings which combine a weight on prior preferences with the implicit preferences inferred from behavioral data. Additionally, we conduct an empirical analysis to demonstrate that models learned with this approach yield comparable gains on click-based performance measures to standard methods with far fewer queries personalized. On three real-world commercial search engine logs, the method leads to substantial reductions in the number of queries re-ranked (2x - 7x fewer queries re-ranked) while maintaining 85-95% of the total gain achieved by the standard approach.","PeriodicalId":440325,"journal":{"name":"Proceedings of the 2015 International Conference on The Theory of Information Retrieval","volume":"161 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126737437","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 5

Transferring Learning To Rank Models for Web Search 将学习转移到网络搜索的排名模型

Proceedings of the 2015 International Conference on The Theory of Information Retrieval Pub Date : 2015-09-27 DOI: 10.1145/2808194.2809463

C. Macdonald, B. T. Dinçer, I. Ounis

{"title":"Transferring Learning To Rank Models for Web Search","authors":"C. Macdonald, B. T. Dinçer, I. Ounis","doi":"10.1145/2808194.2809463","DOIUrl":"https://doi.org/10.1145/2808194.2809463","url":null,"abstract":"Learning to rank techniques provide mechanisms for combining document feature values into learned models that produce effective rankings. However, issues concerning the transferability of learned models between different corpora or subsets of the same corpus are not yet well understood. For instance, is the importance of different feature sets consistent between subsets of a corpus, or whether a learned model obtained on a small subset of the corpus effectively transfer to the larger corpus? By formulating our experiments around two null hypotheses, in this work, we apply a full-factorial experiment design to empirically investigate these questions using the ClueWeb09 and ClueWeb12 corpora, combined with queries from the TREC Web track. Among other observations, our experiments reveal that Clue-Web09 remains an effective choice of training corpus for learning effective models for ClueWeb12, and also that the importance of query independent features varies among the ClueWeb09 and ClueWeb12 corpora. In doing so, this work contributes an important study into the transferability of learning to rank models, as well as empirically-derived best practices for effective retrieval on the ClueWeb12 corpus.","PeriodicalId":440325,"journal":{"name":"Proceedings of the 2015 International Conference on The Theory of Information Retrieval","volume":"14 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129026909","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 8

Theoretical Categorization of Query Performance Predictors 查询性能预测器的理论分类

Proceedings of the 2015 International Conference on The Theory of Information Retrieval Pub Date : 2015-09-27 DOI: 10.1145/2808194.2809475

Victor Makarenkov, Bracha Shapira, L. Rokach

引用次数: 5

Axiomatic Analysis of Smoothing Methods in Language Models for Pseudo-Relevance Feedback 伪相关反馈语言模型平滑方法的公理化分析

Proceedings of the 2015 International Conference on The Theory of Information Retrieval Pub Date : 2015-09-27 DOI: 10.1145/2808194.2809471

Hussein Hazimeh, ChengXiang Zhai

{"title":"Axiomatic Analysis of Smoothing Methods in Language Models for Pseudo-Relevance Feedback","authors":"Hussein Hazimeh, ChengXiang Zhai","doi":"10.1145/2808194.2809471","DOIUrl":"https://doi.org/10.1145/2808194.2809471","url":null,"abstract":"Pseudo-Relevance Feedback (PRF) is an important general technique for improving retrieval effectiveness without requiring any user effort. Several state-of-the-art PRF models are based on the language modeling approach where a query language model is learned based on feedback documents. In all these models, feedback documents are represented with unigram language models smoothed with a collection language model. While collection language model-based smoothing has proven both effective and necessary in using language models for retrieval, we use axiomatic analysis to show that this smoothing scheme inherently causes the feedback model to favor frequent terms and thus violates the IDF constraint needed to ensure selection of discriminative feedback terms. To address this problem, we propose replacing collection language model-based smoothing in the feedback stage with additive smoothing, which is analytically shown to select more discriminative terms. Empirical evaluation further confirms that additive smoothing indeed significantly outperforms collection-based smoothing methods in multiple language model-based PRF models.","PeriodicalId":440325,"journal":{"name":"Proceedings of the 2015 International Conference on The Theory of Information Retrieval","volume":"46 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133199273","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 26

A Relationship between the Average Precision and the Area Under the ROC Curve 平均精度与ROC曲线下面积的关系

Proceedings of the 2015 International Conference on The Theory of Information Retrieval Pub Date : 2015-09-27 DOI: 10.1145/2808194.2809481

Wanhua Su, Yan Yuan, Mu Zhu

引用次数: 94

Condensed List Relevance Models 精简列表关联模型

Proceedings of the 2015 International Conference on The Theory of Information Retrieval Pub Date : 2015-09-27 DOI: 10.1145/2808194.2809491

Fernando Diaz

引用次数: 21