Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval最新文献_第3页

Ensemble Distillation for BERT-Based Ranking Models 基于bert排序模型的集成蒸馏

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval Pub Date : 2021-07-11 DOI: 10.1145/3471158.3472238

Honglei Zhuang, Zhen Qin, Shuguang Han, Xuanhui Wang, Michael Bendersky, Marc Najork

{"title":"Ensemble Distillation for BERT-Based Ranking Models","authors":"Honglei Zhuang, Zhen Qin, Shuguang Han, Xuanhui Wang, Michael Bendersky, Marc Najork","doi":"10.1145/3471158.3472238","DOIUrl":"https://doi.org/10.1145/3471158.3472238","url":null,"abstract":"Over the past two years, large pretrained language models such as BERT have been applied to text ranking problems and showed superior performance on multiple public benchmark data sets. Prior work demonstrated that an ensemble of multiple BERT-based ranking models can not only boost the performance, but also reduce the performance variance. However, an ensemble of models is more costly because it needs computing resource and/or inference time proportional to the number of models. In this paper, we study how to retain the performance of an ensemble of models at the inference cost of a single model by distilling the ensemble into a single BERT-based student ranking model. Specifically, we study different designs of teacher labels, various distillation strategies, as well as multiple distillation losses tailored for ranking problems. We conduct experiments on the MS MARCO passage ranking and the TREC-COVID data set. Our results show that even with these simple distillation techniques, the distilled model can effectively retain the performance gain of the ensemble of multiple models. More interestingly, the performances of distilled models are also more stable than models fine-tuned on original labeled data. The results reveal a promising direction to capitalize on the gains achieved by an ensemble of BERT-based ranking models.","PeriodicalId":289271,"journal":{"name":"Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval","volume":"28 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-07-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122983441","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 13

The Simplest Thing That Can Possibly Work: (Pseudo-)Relevance Feedback via Text Classification 可能奏效的最简单方法：通过文本分类进行（伪）相关性反馈

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval Pub Date : 2021-07-11 DOI: 10.1145/3471158.3472261

Xiao Han, Yuqi Liu, Jimmy Lin

引用次数: 1

ERR is not C/W/L: Exploring the Relationship Between Expected Reciprocal Rank and Other Metrics ERR不是C/W/L:探索期望倒数排名与其他指标之间的关系

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval Pub Date : 2021-07-11 DOI: 10.1145/3471158.3472239

L. Azzopardi, J. Mackenzie, Alistair Moffat

{"title":"ERR is not C/W/L: Exploring the Relationship Between Expected Reciprocal Rank and Other Metrics","authors":"L. Azzopardi, J. Mackenzie, Alistair Moffat","doi":"10.1145/3471158.3472239","DOIUrl":"https://doi.org/10.1145/3471158.3472239","url":null,"abstract":"We explore the relationship between expected reciprocal rank (ERR) and the metrics that are available under the C/W/L framework. On the surface, it appears that the user browsing model associated with ERR can be directly injected into a C/W/L arrangement, to produce system measurements equivalent to those generated from ERR. That assumption is now known to be invalid, and demonstration of the impossibility of ERR being described via C/W/L choices forms the first part of our work. Given that ERR cannot be accommodated within the C/W/L framework, we then explore the extent to which practical use of ERR correlates with metrics that do fit within the C/W/L user browsing model. In this part of the investigation we present a range of shallow-evaluation C/W/L variants that have very high correlation with ERR when compared in experiments involving a large number of TREC runs. That is, while ERR itself is not a C/W/L metric, there are other weighted-precision computations that fit with the user model assumed by C/W/L, and yield system comparisons almost indistinguishable from those generated via the use of ERR.","PeriodicalId":289271,"journal":{"name":"Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval","volume":"77 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-07-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121967110","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 9

Ranking by Language Similarity for Resource Scarce Southern Bantu Languages 资源稀缺的南班图语语言相似度排序

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval Pub Date : 2021-07-11 DOI: 10.1145/3471158.3472251

Catherine Chavula, H. Suleman

{"title":"Ranking by Language Similarity for Resource Scarce Southern Bantu Languages","authors":"Catherine Chavula, H. Suleman","doi":"10.1145/3471158.3472251","DOIUrl":"https://doi.org/10.1145/3471158.3472251","url":null,"abstract":"Resource Scarce Languages (RSLs) lack sufficient resources to use Cross-Lingual Information Retrieval (CLIR) techniques and tools such as machine translation. Consequentially, searching using RSLs is frustrating and usually ends in unsuccessful struggling search. In such search tasks, search engines return low-quality results; relevant documents are either limited and lowly ranked or non-existent. Previous work has shown that alternative relevant results written in similar languages, including dialects, neighbouring and genetically related languages, can assist multilingual RSLs speakers to complete their search tasks. To improve the quality of search results in this context, we propose the re-ranking of documents based on the similarity between the language of the document and the language of the query. Accordingly, we created a dataset of four Southern Bantu languages that includes documents, topics, topical relevance and intelligibility features, and document utility annotations. To understand the intelligibility dimension of the studied languages, we conducted online intelligibility test experiments and used the data for feature selection and intelligibility prediction. We performed re-ranking of search results using offline evaluation, exploring Learning To Rank (LTR). Our results show that integrating topical relevance and intelligibility in ranking slightly improves retrieval effectiveness. Further, results on intelligibility prediction show that classification of intelligibility is feasible at a fair accuracy.","PeriodicalId":289271,"journal":{"name":"Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval","volume":"112 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-07-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126086892","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 2

Incorporating Widget Positioning in Interaction Models of Search Behaviour 在搜索行为的交互模型中加入小部件定位

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval Pub Date : 2021-07-11 DOI: 10.1145/3471158.3472243

Nirmal Roy, A. Câmara, David Maxwell, C. Hauff

{"title":"Incorporating Widget Positioning in Interaction Models of Search Behaviour","authors":"Nirmal Roy, A. Câmara, David Maxwell, C. Hauff","doi":"10.1145/3471158.3472243","DOIUrl":"https://doi.org/10.1145/3471158.3472243","url":null,"abstract":"Models developed to simulate user interactions with search interfaces typically do not consider the visual layout and presentation of a Search Engine Results Page (SERP). In particular, the position and size of interfacewidgets ---such as entity cards and query suggestions---are usually considered a negligible constant. In contrast, in this work, we investigate the impact of widget positioning on user behaviour. To this end, we focus on one specific widget: the Query History Widget (QHW). It allows users to see (and thus reflect) on their recently issued queries. We build a novel simulation model based on Search Economic Theory (SET) that considers how users behave when faced with such a widget by incorporating its positioning on the SERP. We derive five hypotheses from our model and experimentally validate them based on user interaction data gathered for an ad-hoc search task, run across five different placements of the qhw on the SERP. We find partial support for three of the five hypotheses, and indeed observe that a widget's location has a significant impact on search behaviour.","PeriodicalId":289271,"journal":{"name":"Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval","volume":"482 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-07-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133677720","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 5

Session details: Session 3A - Queries 会话详细信息:会话3A -查询

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval Pub Date : 2021-07-11 DOI: 10.1145/3484642

G. Zuccon

引用次数: 0

Evaluating BERT-based Rewards for Question Generation with Reinforcement Learning 用强化学习评估基于bert的问题生成奖励

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval Pub Date : 2021-07-11 DOI: 10.1145/3471158.3472240

Peide Zhu, C. Hauff

引用次数: 4

Session details: Session 1A - Ranking (1) 会议详情:会议1A -排名(1)

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval Pub Date : 2021-07-11 DOI: 10.1145/3484637

Qingyao Ai

引用次数: 0

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval Pub Date : 2021-07-11 DOI: 10.1145/3471158.3472249

Lakshmi Vikraman, Ali Montazeralghaem, Helia Hashemi, W. Bruce Croft, James Allan

{"title":"Passage Similarity and Diversification in Non-factoid Question Answering","authors":"Lakshmi Vikraman, Ali Montazeralghaem, Helia Hashemi, W. Bruce Croft, James Allan","doi":"10.1145/3471158.3472249","DOIUrl":"https://doi.org/10.1145/3471158.3472249","url":null,"abstract":"The rise in popularity of mobile and voice search has led to a shift in focus from document retrieval to short answer passage retrieval for non-factoid questions. Some of the questions have multiple answers, and the aim is to retrieve a set of relevant answer passages, which covers all these alternatives. Compared to documents, answers are more specific and typically form more defined types or groups. Grouping answer passages based on strong similarity measures may provide a means of identifying these types. Typically, kNN clustering in combination with term-based representations have been used in Information Retrieval (IR) scenarios. An alternate method is to use pre-trained distributional representations such as GloVe and BERT, which capture additional semantic relationships. The recent success of trained neural models for various tasks provides the motivation for generating more task-specific representations. However, due to the absence of large datasets for incorporating passage level similarity information, a more feasible alternative is to use weak supervision based training. This information can then be used to generate a final ranked list of diversified answers using standard diversification algorithms. In this paper, we introduce a new dataset NFPassageQA_Sim, with human annotated similarity labels for pairs of answer passages corresponding to each question. These similarity labels are then processed to generate another dataset NFPassageQA_Div, which consists of answer types for these questions. Using the similarity labels, we demonstrate the effectiveness of using weak supervision signals derived from GloVe, fine-tuned and trained using a BERT model for the task of answer passage clustering. Finally, we introduce a model which incorporates these clusters into a MMR (Maximal Marginal Relevance) model, which significantly beats other diversification baselines using both diversity and relevance metrics.","PeriodicalId":289271,"journal":{"name":"Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval","volume":"31 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-07-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131777060","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 3

Decomposition and Interleaving for Variance Reduction of Post-click Metrics 点击后度量方差减少的分解与交错

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval Pub Date : 2021-07-11 DOI: 10.1145/3471158.3472235

Kojiro Iizuka, Yoshifumi Seki, Makoto P. Kato

{"title":"Decomposition and Interleaving for Variance Reduction of Post-click Metrics","authors":"Kojiro Iizuka, Yoshifumi Seki, Makoto P. Kato","doi":"10.1145/3471158.3472235","DOIUrl":"https://doi.org/10.1145/3471158.3472235","url":null,"abstract":"In this study, we propose an efficient method for comparing the post-click metric (e.g., dwell time and conversion rate) of multiple rankings in online experiments. The proposed method involves (1) the decomposition of the post-click metric measurement of a ranking into a click model estimation and a post-click metric measurement of each item in the ranking, and (2) interleaving of multiple rankings to produce a single ranking that preferentially exposes items possessing a high population variance. The decomposition of the post-click metric measurement enables the free layout of items in a ranking and focuses on the measurement of the post-click metric of each item in the multiple rankings. The interleaving of multiple rankings reduces the sample variance of the items possessing a high population variance by optimizing a ranking to be presented to the users so that those items received more samples of the post-click metric. In addition, we provide a proof that the proposed method leads to the minimization of the evaluation error in the ranking comparison and propose two practical techniques to stabilize the online experiment. We performed a comprehensive simulation experiment and a real service setting experiment. The experimental results revealed that (1) the proposed method outperformed existing methods in terms of efficiency and accuracy, and the performance was especially remarkable when the input rankings shared many items, and (2) the two stabilization techniques successfully improved the evaluation accuracy and efficiency. The real service data from our experiments can be used to validate our experiments as well as investigate the user modeling and bandit algorithms.","PeriodicalId":289271,"journal":{"name":"Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval","volume":"43 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-07-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131461004","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 3