元素排序对 LM Agent 性能的影响

Wayne Chi, Ameet Talwalkar, Chris Donahue
{"title":"元素排序对 LM Agent 性能的影响","authors":"Wayne Chi, Ameet Talwalkar, Chris Donahue","doi":"arxiv-2409.12089","DOIUrl":null,"url":null,"abstract":"There has been a surge of interest in language model agents that can navigate\nvirtual environments such as the web or desktop. To navigate such environments,\nagents benefit from information on the various elements (e.g., buttons, text,\nor images) present. It remains unclear which element attributes have the\ngreatest impact on agent performance, especially in environments that only\nprovide a graphical representation (i.e., pixels). Here we find that the\nordering in which elements are presented to the language model is surprisingly\nimpactful--randomizing element ordering in a webpage degrades agent performance\ncomparably to removing all visible text from an agent's state representation.\nWhile a webpage provides a hierarchical ordering of elements, there is no such\nordering when parsing elements directly from pixels. Moreover, as tasks become\nmore challenging and models more sophisticated, our experiments suggest that\nthe impact of ordering increases. Finding an effective ordering is non-trivial.\nWe investigate the impact of various element ordering methods in web and\ndesktop environments. We find that dimensionality reduction provides a viable\nordering for pixel-only environments. We train a UI element detection model to\nderive elements from pixels and apply our findings to an agent\nbenchmark--OmniACT--where we only have access to pixels. Our method completes\nmore than two times as many tasks on average relative to the previous\nstate-of-the-art.","PeriodicalId":501301,"journal":{"name":"arXiv - CS - Machine Learning","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"The Impact of Element Ordering on LM Agent Performance\",\"authors\":\"Wayne Chi, Ameet Talwalkar, Chris Donahue\",\"doi\":\"arxiv-2409.12089\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"There has been a surge of interest in language model agents that can navigate\\nvirtual environments such as the web or desktop. To navigate such environments,\\nagents benefit from information on the various elements (e.g., buttons, text,\\nor images) present. It remains unclear which element attributes have the\\ngreatest impact on agent performance, especially in environments that only\\nprovide a graphical representation (i.e., pixels). Here we find that the\\nordering in which elements are presented to the language model is surprisingly\\nimpactful--randomizing element ordering in a webpage degrades agent performance\\ncomparably to removing all visible text from an agent's state representation.\\nWhile a webpage provides a hierarchical ordering of elements, there is no such\\nordering when parsing elements directly from pixels. Moreover, as tasks become\\nmore challenging and models more sophisticated, our experiments suggest that\\nthe impact of ordering increases. Finding an effective ordering is non-trivial.\\nWe investigate the impact of various element ordering methods in web and\\ndesktop environments. We find that dimensionality reduction provides a viable\\nordering for pixel-only environments. We train a UI element detection model to\\nderive elements from pixels and apply our findings to an agent\\nbenchmark--OmniACT--where we only have access to pixels. Our method completes\\nmore than two times as many tasks on average relative to the previous\\nstate-of-the-art.\",\"PeriodicalId\":501301,\"journal\":{\"name\":\"arXiv - CS - Machine Learning\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-09-18\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"arXiv - CS - Machine Learning\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/arxiv-2409.12089\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Machine Learning","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.12089","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

摘要

人们对能够浏览虚拟环境(如网络或桌面)的语言模型代理兴趣浓厚。要浏览这些环境,代理需要了解各种元素(如按钮、文本或图像)的信息。目前还不清楚哪些元素属性对代理性能的影响最大,尤其是在只提供图形表示(即像素)的环境中。在这里,我们发现元素呈现给语言模型的排序具有令人惊讶的影响--在网页中对元素进行随机排序会降低代理的性能,其程度相当于从代理的状态表示中移除所有可见文本。此外,随着任务越来越具有挑战性,模型越来越复杂,我们的实验表明,排序的影响也在增加。我们研究了网络和桌面环境中各种元素排序方法的影响。我们发现,降维为纯像素环境提供了一种可行的排序方法。我们训练了一个用户界面元素检测模型,以便从像素中提取元素,并将我们的研究结果应用于一个代理基准--OmniACT--其中我们只能访问像素。我们的方法平均完成的任务量是之前最先进方法的两倍多。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
The Impact of Element Ordering on LM Agent Performance
There has been a surge of interest in language model agents that can navigate virtual environments such as the web or desktop. To navigate such environments, agents benefit from information on the various elements (e.g., buttons, text, or images) present. It remains unclear which element attributes have the greatest impact on agent performance, especially in environments that only provide a graphical representation (i.e., pixels). Here we find that the ordering in which elements are presented to the language model is surprisingly impactful--randomizing element ordering in a webpage degrades agent performance comparably to removing all visible text from an agent's state representation. While a webpage provides a hierarchical ordering of elements, there is no such ordering when parsing elements directly from pixels. Moreover, as tasks become more challenging and models more sophisticated, our experiments suggest that the impact of ordering increases. Finding an effective ordering is non-trivial. We investigate the impact of various element ordering methods in web and desktop environments. We find that dimensionality reduction provides a viable ordering for pixel-only environments. We train a UI element detection model to derive elements from pixels and apply our findings to an agent benchmark--OmniACT--where we only have access to pixels. Our method completes more than two times as many tasks on average relative to the previous state-of-the-art.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信