{"title":"元素排序对 LM Agent 性能的影响","authors":"Wayne Chi, Ameet Talwalkar, Chris Donahue","doi":"arxiv-2409.12089","DOIUrl":null,"url":null,"abstract":"There has been a surge of interest in language model agents that can navigate\nvirtual environments such as the web or desktop. To navigate such environments,\nagents benefit from information on the various elements (e.g., buttons, text,\nor images) present. It remains unclear which element attributes have the\ngreatest impact on agent performance, especially in environments that only\nprovide a graphical representation (i.e., pixels). Here we find that the\nordering in which elements are presented to the language model is surprisingly\nimpactful--randomizing element ordering in a webpage degrades agent performance\ncomparably to removing all visible text from an agent's state representation.\nWhile a webpage provides a hierarchical ordering of elements, there is no such\nordering when parsing elements directly from pixels. Moreover, as tasks become\nmore challenging and models more sophisticated, our experiments suggest that\nthe impact of ordering increases. Finding an effective ordering is non-trivial.\nWe investigate the impact of various element ordering methods in web and\ndesktop environments. We find that dimensionality reduction provides a viable\nordering for pixel-only environments. We train a UI element detection model to\nderive elements from pixels and apply our findings to an agent\nbenchmark--OmniACT--where we only have access to pixels. Our method completes\nmore than two times as many tasks on average relative to the previous\nstate-of-the-art.","PeriodicalId":501301,"journal":{"name":"arXiv - CS - Machine Learning","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"The Impact of Element Ordering on LM Agent Performance\",\"authors\":\"Wayne Chi, Ameet Talwalkar, Chris Donahue\",\"doi\":\"arxiv-2409.12089\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"There has been a surge of interest in language model agents that can navigate\\nvirtual environments such as the web or desktop. To navigate such environments,\\nagents benefit from information on the various elements (e.g., buttons, text,\\nor images) present. It remains unclear which element attributes have the\\ngreatest impact on agent performance, especially in environments that only\\nprovide a graphical representation (i.e., pixels). Here we find that the\\nordering in which elements are presented to the language model is surprisingly\\nimpactful--randomizing element ordering in a webpage degrades agent performance\\ncomparably to removing all visible text from an agent's state representation.\\nWhile a webpage provides a hierarchical ordering of elements, there is no such\\nordering when parsing elements directly from pixels. Moreover, as tasks become\\nmore challenging and models more sophisticated, our experiments suggest that\\nthe impact of ordering increases. Finding an effective ordering is non-trivial.\\nWe investigate the impact of various element ordering methods in web and\\ndesktop environments. We find that dimensionality reduction provides a viable\\nordering for pixel-only environments. We train a UI element detection model to\\nderive elements from pixels and apply our findings to an agent\\nbenchmark--OmniACT--where we only have access to pixels. Our method completes\\nmore than two times as many tasks on average relative to the previous\\nstate-of-the-art.\",\"PeriodicalId\":501301,\"journal\":{\"name\":\"arXiv - CS - Machine Learning\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-09-18\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"arXiv - CS - Machine Learning\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/arxiv-2409.12089\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Machine Learning","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.12089","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
The Impact of Element Ordering on LM Agent Performance
There has been a surge of interest in language model agents that can navigate
virtual environments such as the web or desktop. To navigate such environments,
agents benefit from information on the various elements (e.g., buttons, text,
or images) present. It remains unclear which element attributes have the
greatest impact on agent performance, especially in environments that only
provide a graphical representation (i.e., pixels). Here we find that the
ordering in which elements are presented to the language model is surprisingly
impactful--randomizing element ordering in a webpage degrades agent performance
comparably to removing all visible text from an agent's state representation.
While a webpage provides a hierarchical ordering of elements, there is no such
ordering when parsing elements directly from pixels. Moreover, as tasks become
more challenging and models more sophisticated, our experiments suggest that
the impact of ordering increases. Finding an effective ordering is non-trivial.
We investigate the impact of various element ordering methods in web and
desktop environments. We find that dimensionality reduction provides a viable
ordering for pixel-only environments. We train a UI element detection model to
derive elements from pixels and apply our findings to an agent
benchmark--OmniACT--where we only have access to pixels. Our method completes
more than two times as many tasks on average relative to the previous
state-of-the-art.