自动驾驶汽车黑盒测试中的边界依附与探索策略

John M. Thompson, Quentin Goss, M. Akbaş
{"title":"自动驾驶汽车黑盒测试中的边界依附与探索策略","authors":"John M. Thompson, Quentin Goss, M. Akbaş","doi":"10.1109/MOST57249.2023.00028","DOIUrl":null,"url":null,"abstract":"The validation of artificial intelligence (AI) controlled vehicles is a vexing challenge. Black box testing of decision making in these vehicles has been used to abstract out the inner complexity of the AI. In scenario-based black box testing, the AI is placed within a scenario, and the input space for that scenario is explored. The fundamental metric of “how well tested the system is for that scenario” is based on the input state space coverage. Since most of these spaces have a high number of dimensions, it is critical to sample the state space efficiently and identify performance boundaries for the vehicle under test. In this paper, we propose a boundary adherence approach for autonomous vehicle validation that can explore the boundary between targeted and non-targeted behavior. This paper significantly improves and extends our previous approach that focused on generic black-box testing of AI systems by optimizing the algorithm itself, adding new tools for exploration, and applying the strategy to scenario-based AV testing. We provide an example regression of a scenario which illustrates the ability to model boundaries after they have been explored. Further results on higher dimensions show differing adherence strategies can improve exploration efficiency and how boundary exploration focuses on more “interesting” scenarios. Upon exploring the boundary, we found that predictions can be made about whether or not the system will result in targeted behavior.","PeriodicalId":338621,"journal":{"name":"2023 IEEE International Conference on Mobility, Operations, Services and Technologies (MOST)","volume":"34 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2023-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"A Strategy for Boundary Adherence and Exploration in Black-Box Testing of Autonomous Vehicles\",\"authors\":\"John M. Thompson, Quentin Goss, M. Akbaş\",\"doi\":\"10.1109/MOST57249.2023.00028\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"The validation of artificial intelligence (AI) controlled vehicles is a vexing challenge. Black box testing of decision making in these vehicles has been used to abstract out the inner complexity of the AI. In scenario-based black box testing, the AI is placed within a scenario, and the input space for that scenario is explored. The fundamental metric of “how well tested the system is for that scenario” is based on the input state space coverage. Since most of these spaces have a high number of dimensions, it is critical to sample the state space efficiently and identify performance boundaries for the vehicle under test. In this paper, we propose a boundary adherence approach for autonomous vehicle validation that can explore the boundary between targeted and non-targeted behavior. This paper significantly improves and extends our previous approach that focused on generic black-box testing of AI systems by optimizing the algorithm itself, adding new tools for exploration, and applying the strategy to scenario-based AV testing. We provide an example regression of a scenario which illustrates the ability to model boundaries after they have been explored. Further results on higher dimensions show differing adherence strategies can improve exploration efficiency and how boundary exploration focuses on more “interesting” scenarios. Upon exploring the boundary, we found that predictions can be made about whether or not the system will result in targeted behavior.\",\"PeriodicalId\":338621,\"journal\":{\"name\":\"2023 IEEE International Conference on Mobility, Operations, Services and Technologies (MOST)\",\"volume\":\"34 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2023-05-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"2023 IEEE International Conference on Mobility, Operations, Services and Technologies (MOST)\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1109/MOST57249.2023.00028\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"2023 IEEE International Conference on Mobility, Operations, Services and Technologies (MOST)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/MOST57249.2023.00028","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

摘要

人工智能(AI)控制车辆的验证是一项棘手的挑战。这些车辆的决策黑盒测试被用来抽象出人工智能的内在复杂性。在基于场景的黑盒测试中,AI被放置在场景中,并探索该场景的输入空间。“系统在该场景中测试得有多好”的基本度量是基于输入状态空间覆盖的。由于这些空间中的大多数都有大量的维度,因此有效地对状态空间进行采样并识别被测车辆的性能边界至关重要。在本文中,我们提出了一种用于自动驾驶汽车验证的边界粘附方法,该方法可以探索目标和非目标行为之间的边界。本文通过优化算法本身,添加新的探索工具,并将该策略应用于基于场景的自动驾驶测试,显著改进和扩展了我们之前专注于人工智能系统的通用黑盒测试的方法。我们提供了一个场景的示例回归,它说明了在探索边界之后对边界进行建模的能力。在更高维度上的进一步结果表明,不同的粘附策略可以提高勘探效率,以及边界勘探如何关注更“有趣”的场景。在探索边界时,我们发现可以预测系统是否会导致目标行为。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
A Strategy for Boundary Adherence and Exploration in Black-Box Testing of Autonomous Vehicles
The validation of artificial intelligence (AI) controlled vehicles is a vexing challenge. Black box testing of decision making in these vehicles has been used to abstract out the inner complexity of the AI. In scenario-based black box testing, the AI is placed within a scenario, and the input space for that scenario is explored. The fundamental metric of “how well tested the system is for that scenario” is based on the input state space coverage. Since most of these spaces have a high number of dimensions, it is critical to sample the state space efficiently and identify performance boundaries for the vehicle under test. In this paper, we propose a boundary adherence approach for autonomous vehicle validation that can explore the boundary between targeted and non-targeted behavior. This paper significantly improves and extends our previous approach that focused on generic black-box testing of AI systems by optimizing the algorithm itself, adding new tools for exploration, and applying the strategy to scenario-based AV testing. We provide an example regression of a scenario which illustrates the ability to model boundaries after they have been explored. Further results on higher dimensions show differing adherence strategies can improve exploration efficiency and how boundary exploration focuses on more “interesting” scenarios. Upon exploring the boundary, we found that predictions can be made about whether or not the system will result in targeted behavior.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信