{"title":"了解百度-ULTR 日志政策对双塔模型的影响","authors":"Morris de Haan, Philipp Hager","doi":"arxiv-2409.12043","DOIUrl":null,"url":null,"abstract":"Despite the popularity of the two-tower model for unbiased learning to rank\n(ULTR) tasks, recent work suggests that it suffers from a major limitation that\ncould lead to its collapse in industry applications: the problem of logging\npolicy confounding. Several potential solutions have even been proposed;\nhowever, the evaluation of these methods was mostly conducted using\nsemi-synthetic simulation experiments. This paper bridges the gap between\ntheory and practice by investigating the confounding problem on the largest\nreal-world dataset, Baidu-ULTR. Our main contributions are threefold: 1) we\nshow that the conditions for the confounding problem are given on Baidu-ULTR,\n2) the confounding problem bears no significant effect on the two-tower model,\nand 3) we point to a potential mismatch between expert annotations, the golden\nstandard in ULTR, and user click behavior.","PeriodicalId":501281,"journal":{"name":"arXiv - CS - Information Retrieval","volume":"26 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Understanding the Effects of the Baidu-ULTR Logging Policy on Two-Tower Models\",\"authors\":\"Morris de Haan, Philipp Hager\",\"doi\":\"arxiv-2409.12043\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Despite the popularity of the two-tower model for unbiased learning to rank\\n(ULTR) tasks, recent work suggests that it suffers from a major limitation that\\ncould lead to its collapse in industry applications: the problem of logging\\npolicy confounding. Several potential solutions have even been proposed;\\nhowever, the evaluation of these methods was mostly conducted using\\nsemi-synthetic simulation experiments. This paper bridges the gap between\\ntheory and practice by investigating the confounding problem on the largest\\nreal-world dataset, Baidu-ULTR. Our main contributions are threefold: 1) we\\nshow that the conditions for the confounding problem are given on Baidu-ULTR,\\n2) the confounding problem bears no significant effect on the two-tower model,\\nand 3) we point to a potential mismatch between expert annotations, the golden\\nstandard in ULTR, and user click behavior.\",\"PeriodicalId\":501281,\"journal\":{\"name\":\"arXiv - CS - Information Retrieval\",\"volume\":\"26 1\",\"pages\":\"\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-09-18\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"arXiv - CS - Information Retrieval\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/arxiv-2409.12043\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Information Retrieval","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.12043","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
Understanding the Effects of the Baidu-ULTR Logging Policy on Two-Tower Models
Despite the popularity of the two-tower model for unbiased learning to rank
(ULTR) tasks, recent work suggests that it suffers from a major limitation that
could lead to its collapse in industry applications: the problem of logging
policy confounding. Several potential solutions have even been proposed;
however, the evaluation of these methods was mostly conducted using
semi-synthetic simulation experiments. This paper bridges the gap between
theory and practice by investigating the confounding problem on the largest
real-world dataset, Baidu-ULTR. Our main contributions are threefold: 1) we
show that the conditions for the confounding problem are given on Baidu-ULTR,
2) the confounding problem bears no significant effect on the two-tower model,
and 3) we point to a potential mismatch between expert annotations, the golden
standard in ULTR, and user click behavior.