Understanding the Effects of the Baidu-ULTR Logging Policy on Two-Tower Models

Morris de Haan, Philipp Hager
{"title":"Understanding the Effects of the Baidu-ULTR Logging Policy on Two-Tower Models","authors":"Morris de Haan, Philipp Hager","doi":"arxiv-2409.12043","DOIUrl":null,"url":null,"abstract":"Despite the popularity of the two-tower model for unbiased learning to rank\n(ULTR) tasks, recent work suggests that it suffers from a major limitation that\ncould lead to its collapse in industry applications: the problem of logging\npolicy confounding. Several potential solutions have even been proposed;\nhowever, the evaluation of these methods was mostly conducted using\nsemi-synthetic simulation experiments. This paper bridges the gap between\ntheory and practice by investigating the confounding problem on the largest\nreal-world dataset, Baidu-ULTR. Our main contributions are threefold: 1) we\nshow that the conditions for the confounding problem are given on Baidu-ULTR,\n2) the confounding problem bears no significant effect on the two-tower model,\nand 3) we point to a potential mismatch between expert annotations, the golden\nstandard in ULTR, and user click behavior.","PeriodicalId":501281,"journal":{"name":"arXiv - CS - Information Retrieval","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Information Retrieval","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.12043","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

Despite the popularity of the two-tower model for unbiased learning to rank (ULTR) tasks, recent work suggests that it suffers from a major limitation that could lead to its collapse in industry applications: the problem of logging policy confounding. Several potential solutions have even been proposed; however, the evaluation of these methods was mostly conducted using semi-synthetic simulation experiments. This paper bridges the gap between theory and practice by investigating the confounding problem on the largest real-world dataset, Baidu-ULTR. Our main contributions are threefold: 1) we show that the conditions for the confounding problem are given on Baidu-ULTR, 2) the confounding problem bears no significant effect on the two-tower model, and 3) we point to a potential mismatch between expert annotations, the golden standard in ULTR, and user click behavior.
了解百度-ULTR 日志政策对双塔模型的影响
尽管双塔模型在无偏学习排名(ULTR)任务中很受欢迎,但最近的研究表明,它存在一个可能导致其在行业应用中崩溃的主要局限性:记录政策混淆问题。人们甚至提出了几种潜在的解决方案;然而,对这些方法的评估大多是通过半合成模拟实验进行的。本文通过在最大的真实世界数据集百度-ULTR 上研究混淆问题,弥补了理论与实践之间的差距。我们的主要贡献有三个方面:1)我们证明了在百度-ULTR 上混淆问题的条件;2)混淆问题对双塔模型没有显著影响;3)我们指出了专家注释(ULTR 的黄金标准)与用户点击行为之间潜在的不匹配。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信