What is the Right Notion of Distance between Predict-then-Optimize Tasks?

Paula Rodriguez-Diaz, Lingkai Kong, Kai Wang, David Alvarez-Melis, Milind Tambe
{"title":"What is the Right Notion of Distance between Predict-then-Optimize Tasks?","authors":"Paula Rodriguez-Diaz, Lingkai Kong, Kai Wang, David Alvarez-Melis, Milind Tambe","doi":"arxiv-2409.06997","DOIUrl":null,"url":null,"abstract":"Comparing datasets is a fundamental task in machine learning, essential for\nvarious learning paradigms; from evaluating train and test datasets for model\ngeneralization to using dataset similarity for detecting data drift. While\ntraditional notions of dataset distances offer principled measures of\nsimilarity, their utility has largely been assessed through prediction error\nminimization. However, in Predict-then-Optimize (PtO) frameworks, where\npredictions serve as inputs for downstream optimization tasks, model\nperformance is measured through decision regret minimization rather than\nprediction error minimization. In this work, we (i) show that traditional\ndataset distances, which rely solely on feature and label dimensions, lack\ninformativeness in the PtO context, and (ii) propose a new dataset distance\nthat incorporates the impacts of downstream decisions. Our results show that\nthis decision-aware dataset distance effectively captures adaptation success in\nPtO contexts, providing a PtO adaptation bound in terms of dataset distance.\nEmpirically, we show that our proposed distance measure accurately predicts\ntransferability across three different PtO tasks from the literature.","PeriodicalId":501301,"journal":{"name":"arXiv - CS - Machine Learning","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-09-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Machine Learning","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.06997","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

Comparing datasets is a fundamental task in machine learning, essential for various learning paradigms; from evaluating train and test datasets for model generalization to using dataset similarity for detecting data drift. While traditional notions of dataset distances offer principled measures of similarity, their utility has largely been assessed through prediction error minimization. However, in Predict-then-Optimize (PtO) frameworks, where predictions serve as inputs for downstream optimization tasks, model performance is measured through decision regret minimization rather than prediction error minimization. In this work, we (i) show that traditional dataset distances, which rely solely on feature and label dimensions, lack informativeness in the PtO context, and (ii) propose a new dataset distance that incorporates the impacts of downstream decisions. Our results show that this decision-aware dataset distance effectively captures adaptation success in PtO contexts, providing a PtO adaptation bound in terms of dataset distance. Empirically, we show that our proposed distance measure accurately predicts transferability across three different PtO tasks from the literature.
什么是 "先预测后优化 "任务间距离的正确概念?
比较数据集是机器学习的一项基本任务,对各种学习范式都至关重要;从评估训练数据集和测试数据集以实现模型泛化,到利用数据集相似性检测数据漂移,不一而足。虽然传统的数据集距离概念提供了原则性的相似性度量,但其效用主要是通过预测错误最小化来评估的。然而,在预测-优化(PtO)框架中,预测是下游优化任务的输入,模型性能是通过决策遗憾最小化而不是预测误差最小化来衡量的。在这项工作中,我们(i) 证明了仅依赖于特征和标签维度的传统数据集距离在 PtO 环境中缺乏信息性,(ii) 提出了一种新的数据集距离,它包含了下游决策的影响。我们的研究结果表明,这种决策感知数据集距离能有效捕捉 PtO 情境下的适应成功率,并提供了数据集距离的 PtO 适应约束。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信