2021 IEEE International Conference on Artificial Intelligence Testing (AITest)最新文献

筛选
英文 中文
An Industrial Workbench for Test Scenario Identification for Autonomous Driving Software 自动驾驶软件测试场景识别的工业工作台
2021 IEEE International Conference on Artificial Intelligence Testing (AITest) Pub Date : 2021-08-01 DOI: 10.1109/AITEST52744.2021.00024
Qunying Song, Kai Tan, P. Runeson, Stefan Persson
{"title":"An Industrial Workbench for Test Scenario Identification for Autonomous Driving Software","authors":"Qunying Song, Kai Tan, P. Runeson, Stefan Persson","doi":"10.1109/AITEST52744.2021.00024","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00024","url":null,"abstract":"Testing of autonomous vehicles involves enormous challenges for the automotive industry. The number of real-world driving scenarios is extremely large, and choosing effective test scenarios is essential, as well as combining simulated and real world testing. We present an industrial workbench of tools and workflows to generate efficient and effective test scenarios for active safety and autonomous driving functions. The workbench is based on existing engineering tools, and helps smoothly integrate simulated testing, with real vehicle parameters and software. We aim to validate the workbench with real cases and further refine the input model parameters and distributions.","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"122 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134025638","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 4
Critical and Challenging Scenario Generation based on Automatic Action Behavior Sequence Optimization: 2021 IEEE Autonomous Driving AI Test Challenge Group 108 基于自动动作行为序列优化的关键和具有挑战性的场景生成:2021 IEEE自动驾驶AI测试挑战组108
2021 IEEE International Conference on Artificial Intelligence Testing (AITest) Pub Date : 2021-08-01 DOI: 10.1109/AITEST52744.2021.00032
David Kaufmann, Lorenz Klampfl, Florian Klück, Martin Zimmermann, Jianbo Tao
{"title":"Critical and Challenging Scenario Generation based on Automatic Action Behavior Sequence Optimization: 2021 IEEE Autonomous Driving AI Test Challenge Group 108","authors":"David Kaufmann, Lorenz Klampfl, Florian Klück, Martin Zimmermann, Jianbo Tao","doi":"10.1109/AITEST52744.2021.00032","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00032","url":null,"abstract":"Assuring the safety of automated and autonomous driving functions is crucial for a safe deployment of self-driving vehicles on public roads. This includes the need for automated virtual testing methods and exhaustive search for critical scenarios that potentially reveal faults in the driving feature under test. In the past, researches have demonstrated the effectiveness of search-based testing to create situations that result in unintended behavior of the driving feature. In this paper, we contribute to this field of research by developing a method for automated generation of diverse critical scenarios based on a search algorithm that iterative optimizes behavior action sequences of the surrounding traffic participants towards critical situations. Utilizing the provided LG SVL Simulator pipeline, our method effectively generated both critical and challenging test scenarios that either revealed faulty behavior of the ego-vehicle (crash or near-crash) or showed extraordinary behavior of the surrounding traffic participants (e.g. approaching traffic on wrong lane).","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"102 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131699412","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 5
Towards Trace-Graphs for Data-driven Test Case Mining in the Domain of Automated Driving 面向自动驾驶领域数据驱动测试用例挖掘的跟踪图研究
2021 IEEE International Conference on Artificial Intelligence Testing (AITest) Pub Date : 2021-08-01 DOI: 10.1109/AITEST52744.2021.00019
Noah Metzger, Lars Hoffmann, Christian Bartelt, H. Stuckenschmidt, M. Wommer, Maria Belen Bescos del Castillo
{"title":"Towards Trace-Graphs for Data-driven Test Case Mining in the Domain of Automated Driving","authors":"Noah Metzger, Lars Hoffmann, Christian Bartelt, H. Stuckenschmidt, M. Wommer, Maria Belen Bescos del Castillo","doi":"10.1109/AITEST52744.2021.00019","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00019","url":null,"abstract":"Technological innovations in data science and artificial intelligence research have led to increasing task automation in autonomous driving. Currently, most test engineering processes in the industry are based on manual test case specification and the accumulation of certain test mile thresholds. This test engineering is very cost- and time-consuming. But especially the test case specification and selection is influenced by human bias. A promising potential approach to overcome those drawbacks is data-driven test case mining based on big data of sensor/actor recordings from real driving situations. The key challenge for data-driven test case mining, is handling the huge amount of data generated by the car sensors related to the open-world context of autonomous driving. This paper presents a novel data-driven approach to quantify events and situations which occur during recorded test drives by transforming multivariate time series data into a condensed, directed graph. This graphical model makes such events and situations comparable and enables the application of established coverage criteria and thus supports statistical statements regarding the coverage of potentially infinite operating environments. To illustrate the benefits of this approach, we conducted a quantitative evaluation in an industrial context in the domain of automated driving, including over 1000 km of field data. The overall results prove that a significant amount of the data characteristics do not get lost during the Graph transformation.","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"45 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115146837","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
Heuristic-word-selection Genetic Algorithm for Generating Natural Language Adversarial Examples 用于生成自然语言对抗示例的启发式词选择遗传算法
2021 IEEE International Conference on Artificial Intelligence Testing (AITest) Pub Date : 2021-08-01 DOI: 10.1109/AITEST52744.2021.00018
Shijun Ye, Pengcheng Zhang, Hai Dong, Shunhui Ji
{"title":"Heuristic-word-selection Genetic Algorithm for Generating Natural Language Adversarial Examples","authors":"Shijun Ye, Pengcheng Zhang, Hai Dong, Shunhui Ji","doi":"10.1109/AITEST52744.2021.00018","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00018","url":null,"abstract":"Using adversarial examples as test cases is an effective way to detect the security and reliability problems of Deep Neural Networks (DNNs). Genetic Algorithm (GA) performs well in generating Natural Language Processing (NLP) adversarial examples due to its excellent ability to search for optimal solutions. However, the existing GAs need substantial time to generate NLP adversarial examples. Moreover, the semantics of examples generated by GAs is quite different from real texts, which makes it impossible to generate efficient and reliable adversarial examples. To address these two problems, this paper proposes a Heuristic-word-selection Genetic Algorithm (HGA) to generate NLP adversarial examples. Initial experimental results indicate that HGA not only improves the efficiency of GA to generate adversarial examples, but also enhances the preservation of text semantics, while maintaining the existing GAs' advantages.","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"87 ","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114097602","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Validating Gaussian Process Models with Simulation-Based Calibration 基于仿真标定的高斯过程模型验证
2021 IEEE International Conference on Artificial Intelligence Testing (AITest) Pub Date : 2021-08-01 DOI: 10.1109/AITEST52744.2021.00028
John Mcleod, F. Simpson
{"title":"Validating Gaussian Process Models with Simulation-Based Calibration","authors":"John Mcleod, F. Simpson","doi":"10.1109/AITEST52744.2021.00028","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00028","url":null,"abstract":"Gaussian process priors are a popular choice for Bayesian analysis of regression problems. However, the implementation of these models can be complex, and ensuring that the implementation is correct can be challenging. In this paper we introduce Gaussian process simulation-based calibration, a procedure for validating the implementation of Gaussian process models and demonstrate the efficacy of this procedure in identifying a bug in existing code. We also present a novel application of this procedure to identify when marginalisation of the model hyperparameters is necessary.","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"24 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126898204","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 2
AI robustness analysis with consideration of corner cases 考虑极端情况的人工智能鲁棒性分析
2021 IEEE International Conference on Artificial Intelligence Testing (AITest) Pub Date : 2021-08-01 DOI: 10.1109/AITEST52744.2021.00016
Tinghui Ouyang, Yoshinao Isobe, Vicent Sanz Marco, Jun Ogata, Yoshiki Seo, Y. Oiwa
{"title":"AI robustness analysis with consideration of corner cases","authors":"Tinghui Ouyang, Yoshinao Isobe, Vicent Sanz Marco, Jun Ogata, Yoshiki Seo, Y. Oiwa","doi":"10.1109/AITEST52744.2021.00016","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00016","url":null,"abstract":"Corner cases are a set of high-risk data in deep learning (DL) systems, which could lead to incorrect and unexpected behaviors. To study corner cases' influence on DL models' robustness and stability, this paper implemented a research with corner case description and detection, as well as DL model's robustness measurement. Firstly, a corner case descriptor based on surprise adequacy was introduced for corner case data detection, its high values are proved useful to reflect incorrect behaviors in classification. Then, based on the proposed corner case descriptor, training dataset was updated by removing data having high possibility to be corner case, and utilized for model retraining. A practical robustness analysis method was applied to measure the robustness radius of both the original and the retrained DL models. Through experiments on MNIST data and industrial vision inspection data, an interesting phenomenon is found that, with consideration of corner case data removing in training data, DL models' robustness can be improved to some extent.","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"118 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126967793","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 5
An Empirical Testing of Autonomous Vehicle Simulator System for Urban Driving 城市自动驾驶汽车仿真系统的实证检验
2021 IEEE International Conference on Artificial Intelligence Testing (AITest) Pub Date : 2021-08-01 DOI: 10.1109/AITEST52744.2021.00031
J. Seymour, D. Ho, Q. Luu
{"title":"An Empirical Testing of Autonomous Vehicle Simulator System for Urban Driving","authors":"J. Seymour, D. Ho, Q. Luu","doi":"10.1109/AITEST52744.2021.00031","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00031","url":null,"abstract":"Safety is one of the main challenges that prohibit autonomous vehicles (AV), requiring them to be well tested ahead of being allowed on the road. In comparison with road tests, simulators allow us to validate the AV conveniently and affordably. However, it remains unclear how to best use the AV-based simulator system for testing effectively. Our paper presents an empirical testing of AV simulator system that combines the SVL simulator and the Apollo platform. We propose 576 test cases which are inspired by four naturalistic driving situations with pedestrians and surrounding cars. We found that the SVL can imitate realistic safe and collision situations; and at the same time, Apollo can drive the car quite safely. On the other hand, we noted that the system failed to detect pedestrians or vehicles on the road in three out of four classes, accounting for 10.0% total number of scenarios tested. We further applied metamorphic testing to identify inconsistencies in the system with additional 486 test cases. We then discussed some insights into the scenarios that may cause hazardous situations in real life. In summary, this paper provides a new empirical evidence to strengthen the assertion that the simulator-based system can be an indispensable tool for a comprehensive testing of the AV.","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"192 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127667982","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 13
SALVO: Automated Generation of Diversified Tests for Self-driving Cars from Existing Maps SALVO:根据现有地图自动生成自动驾驶汽车的多种测试
2021 IEEE International Conference on Artificial Intelligence Testing (AITest) Pub Date : 2021-08-01 DOI: 10.1109/AITEST52744.2021.00033
Vuong Nguyen, Stefan Huber, Alessio Gambi
{"title":"SALVO: Automated Generation of Diversified Tests for Self-driving Cars from Existing Maps","authors":"Vuong Nguyen, Stefan Huber, Alessio Gambi","doi":"10.1109/AITEST52744.2021.00033","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00033","url":null,"abstract":"Simulation-based tests are more cost-effective and less dangerous than field tests; hence, they are becoming the norm for thoroughly testing self-driving cars in virtual environments. High-quality simulation-based testing requires physically accurate computer simulations, detailed and photo-realistic maps, and systematic approaches for generating tests. Moreover, since creating detailed maps is a manual process, they are expensive, and testers should avoid wasting such valuable resources, for example, by generating irrelevant test cases. To address those issues, we propose SALVO a fully automated approach to identify quantifiably diverse and critical driving scenarios that can be instantiated on existing high-definition maps and implement them in an industrial driving simulator as executable test cases. The evaluation of SALVO in the context of the 2021 IEEE Autonomous Driving AI Test Challenge showed that it could analyze maps of different complexity and identify many critical driving scenarios in minutes. Furthermore, the tests SALVO generated stressed a state-of-art self-driving car software in quantifiably diverse ways and exposed issues with its implementation.","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"118 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116375324","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 16
Quantifying Nondeterminism and Inconsistency in Self-organizing Map Implementations 量化自组织映射实现中的不确定性和不一致性
2021 IEEE International Conference on Artificial Intelligence Testing (AITest) Pub Date : 2021-08-01 DOI: 10.1109/AITEST52744.2021.00026
Sydur Rahaman, Raina Samuel, Iulian Neamtiu
{"title":"Quantifying Nondeterminism and Inconsistency in Self-organizing Map Implementations","authors":"Sydur Rahaman, Raina Samuel, Iulian Neamtiu","doi":"10.1109/AITEST52744.2021.00026","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00026","url":null,"abstract":"Self-organizing maps (SOMs) are a popular approach for neural network-based unsupervised learning. However the reliability of self-organizing map implementations has not been investigated. Using internal and external metrics, we define and check two basic SOM properties. First, determinism: a given SOM implementation should produce the same SOM when run repeatedly on the same training dataset. Second, consistency: two SOM implementations should produce similar SOMs when presented with the same training dataset. We check these properties in four popular SOM implementations. We ran our approach on 381 popular datasets used in health, medicine, and other critical domains. We found that implementations violate these basic properties. For example, 375 out of 381 datasets have nondeterministic outcomes; for 51–92% of datasets, toolkits yield significantly different SOM clusterings; and clustering accuracy might be so inconsistent as to vary by a factor of four between toolkits. This undermines SOM reliability, and the reliability of results obtained via SOMs. Our study shines a light on what to expect, in practice, when running actual SOM implementations. Our findings suggest that for critical applications, SOM users should not take reliability for granted; rather, multiple runs and different toolkits should be considered and compared.","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"361 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122779700","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 3
Evolutionary Generation of Test Case for Deep Neural Network Based on Coverage Guidance 基于覆盖引导的深度神经网络测试用例进化生成
2021 IEEE International Conference on Artificial Intelligence Testing (AITest) Pub Date : 2021-08-01 DOI: 10.1109/AITEST52744.2021.00014
Liming Hu, Shunhui Ji, Qiyin Dai, Pengcheng Zhang
{"title":"Evolutionary Generation of Test Case for Deep Neural Network Based on Coverage Guidance","authors":"Liming Hu, Shunhui Ji, Qiyin Dai, Pengcheng Zhang","doi":"10.1109/AITEST52744.2021.00014","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00014","url":null,"abstract":"In the past few years, Deep Neural Network(DNN) has made great progress. However, it is difficult to guarantee that DNN-based applications can get satisfactory results. Testing is an effective technology to improve the accuracy and robustness of DNN, in which test case generation is an important task. In this paper, we combine the differential evolution algorithm and coverage criterion which is stronger than neuron coverage to generate test cases for DNN. This method uses the prediction loss of DNN and coverage criterion as the fitness function to generate small perturbations at the pixel level between different channels of the image. Then the generated perturbations are added to the image to construct a test case. Theoretically, the test cases generated by this method can not only make the neural network produce misclassification, but also achieve higher coverage for neurons in the DNN, so that defects in DNN can be detected more comprehensively.","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"31 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116169305","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
相关产品
×
本文献相关产品
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信
小红书