{"title":"An Industrial Workbench for Test Scenario Identification for Autonomous Driving Software","authors":"Qunying Song, Kai Tan, P. Runeson, Stefan Persson","doi":"10.1109/AITEST52744.2021.00024","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00024","url":null,"abstract":"Testing of autonomous vehicles involves enormous challenges for the automotive industry. The number of real-world driving scenarios is extremely large, and choosing effective test scenarios is essential, as well as combining simulated and real world testing. We present an industrial workbench of tools and workflows to generate efficient and effective test scenarios for active safety and autonomous driving functions. The workbench is based on existing engineering tools, and helps smoothly integrate simulated testing, with real vehicle parameters and software. We aim to validate the workbench with real cases and further refine the input model parameters and distributions.","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"122 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134025638","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
David Kaufmann, Lorenz Klampfl, Florian Klück, Martin Zimmermann, Jianbo Tao
{"title":"Critical and Challenging Scenario Generation based on Automatic Action Behavior Sequence Optimization: 2021 IEEE Autonomous Driving AI Test Challenge Group 108","authors":"David Kaufmann, Lorenz Klampfl, Florian Klück, Martin Zimmermann, Jianbo Tao","doi":"10.1109/AITEST52744.2021.00032","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00032","url":null,"abstract":"Assuring the safety of automated and autonomous driving functions is crucial for a safe deployment of self-driving vehicles on public roads. This includes the need for automated virtual testing methods and exhaustive search for critical scenarios that potentially reveal faults in the driving feature under test. In the past, researches have demonstrated the effectiveness of search-based testing to create situations that result in unintended behavior of the driving feature. In this paper, we contribute to this field of research by developing a method for automated generation of diverse critical scenarios based on a search algorithm that iterative optimizes behavior action sequences of the surrounding traffic participants towards critical situations. Utilizing the provided LG SVL Simulator pipeline, our method effectively generated both critical and challenging test scenarios that either revealed faulty behavior of the ego-vehicle (crash or near-crash) or showed extraordinary behavior of the surrounding traffic participants (e.g. approaching traffic on wrong lane).","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"102 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131699412","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Noah Metzger, Lars Hoffmann, Christian Bartelt, H. Stuckenschmidt, M. Wommer, Maria Belen Bescos del Castillo
{"title":"Towards Trace-Graphs for Data-driven Test Case Mining in the Domain of Automated Driving","authors":"Noah Metzger, Lars Hoffmann, Christian Bartelt, H. Stuckenschmidt, M. Wommer, Maria Belen Bescos del Castillo","doi":"10.1109/AITEST52744.2021.00019","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00019","url":null,"abstract":"Technological innovations in data science and artificial intelligence research have led to increasing task automation in autonomous driving. Currently, most test engineering processes in the industry are based on manual test case specification and the accumulation of certain test mile thresholds. This test engineering is very cost- and time-consuming. But especially the test case specification and selection is influenced by human bias. A promising potential approach to overcome those drawbacks is data-driven test case mining based on big data of sensor/actor recordings from real driving situations. The key challenge for data-driven test case mining, is handling the huge amount of data generated by the car sensors related to the open-world context of autonomous driving. This paper presents a novel data-driven approach to quantify events and situations which occur during recorded test drives by transforming multivariate time series data into a condensed, directed graph. This graphical model makes such events and situations comparable and enables the application of established coverage criteria and thus supports statistical statements regarding the coverage of potentially infinite operating environments. To illustrate the benefits of this approach, we conducted a quantitative evaluation in an industrial context in the domain of automated driving, including over 1000 km of field data. The overall results prove that a significant amount of the data characteristics do not get lost during the Graph transformation.","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"45 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115146837","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Heuristic-word-selection Genetic Algorithm for Generating Natural Language Adversarial Examples","authors":"Shijun Ye, Pengcheng Zhang, Hai Dong, Shunhui Ji","doi":"10.1109/AITEST52744.2021.00018","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00018","url":null,"abstract":"Using adversarial examples as test cases is an effective way to detect the security and reliability problems of Deep Neural Networks (DNNs). Genetic Algorithm (GA) performs well in generating Natural Language Processing (NLP) adversarial examples due to its excellent ability to search for optimal solutions. However, the existing GAs need substantial time to generate NLP adversarial examples. Moreover, the semantics of examples generated by GAs is quite different from real texts, which makes it impossible to generate efficient and reliable adversarial examples. To address these two problems, this paper proposes a Heuristic-word-selection Genetic Algorithm (HGA) to generate NLP adversarial examples. Initial experimental results indicate that HGA not only improves the efficiency of GA to generate adversarial examples, but also enhances the preservation of text semantics, while maintaining the existing GAs' advantages.","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"87 ","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114097602","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Validating Gaussian Process Models with Simulation-Based Calibration","authors":"John Mcleod, F. Simpson","doi":"10.1109/AITEST52744.2021.00028","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00028","url":null,"abstract":"Gaussian process priors are a popular choice for Bayesian analysis of regression problems. However, the implementation of these models can be complex, and ensuring that the implementation is correct can be challenging. In this paper we introduce Gaussian process simulation-based calibration, a procedure for validating the implementation of Gaussian process models and demonstrate the efficacy of this procedure in identifying a bug in existing code. We also present a novel application of this procedure to identify when marginalisation of the model hyperparameters is necessary.","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"24 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126898204","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Tinghui Ouyang, Yoshinao Isobe, Vicent Sanz Marco, Jun Ogata, Yoshiki Seo, Y. Oiwa
{"title":"AI robustness analysis with consideration of corner cases","authors":"Tinghui Ouyang, Yoshinao Isobe, Vicent Sanz Marco, Jun Ogata, Yoshiki Seo, Y. Oiwa","doi":"10.1109/AITEST52744.2021.00016","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00016","url":null,"abstract":"Corner cases are a set of high-risk data in deep learning (DL) systems, which could lead to incorrect and unexpected behaviors. To study corner cases' influence on DL models' robustness and stability, this paper implemented a research with corner case description and detection, as well as DL model's robustness measurement. Firstly, a corner case descriptor based on surprise adequacy was introduced for corner case data detection, its high values are proved useful to reflect incorrect behaviors in classification. Then, based on the proposed corner case descriptor, training dataset was updated by removing data having high possibility to be corner case, and utilized for model retraining. A practical robustness analysis method was applied to measure the robustness radius of both the original and the retrained DL models. Through experiments on MNIST data and industrial vision inspection data, an interesting phenomenon is found that, with consideration of corner case data removing in training data, DL models' robustness can be improved to some extent.","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"118 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126967793","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"An Empirical Testing of Autonomous Vehicle Simulator System for Urban Driving","authors":"J. Seymour, D. Ho, Q. Luu","doi":"10.1109/AITEST52744.2021.00031","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00031","url":null,"abstract":"Safety is one of the main challenges that prohibit autonomous vehicles (AV), requiring them to be well tested ahead of being allowed on the road. In comparison with road tests, simulators allow us to validate the AV conveniently and affordably. However, it remains unclear how to best use the AV-based simulator system for testing effectively. Our paper presents an empirical testing of AV simulator system that combines the SVL simulator and the Apollo platform. We propose 576 test cases which are inspired by four naturalistic driving situations with pedestrians and surrounding cars. We found that the SVL can imitate realistic safe and collision situations; and at the same time, Apollo can drive the car quite safely. On the other hand, we noted that the system failed to detect pedestrians or vehicles on the road in three out of four classes, accounting for 10.0% total number of scenarios tested. We further applied metamorphic testing to identify inconsistencies in the system with additional 486 test cases. We then discussed some insights into the scenarios that may cause hazardous situations in real life. In summary, this paper provides a new empirical evidence to strengthen the assertion that the simulator-based system can be an indispensable tool for a comprehensive testing of the AV.","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"192 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127667982","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"SALVO: Automated Generation of Diversified Tests for Self-driving Cars from Existing Maps","authors":"Vuong Nguyen, Stefan Huber, Alessio Gambi","doi":"10.1109/AITEST52744.2021.00033","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00033","url":null,"abstract":"Simulation-based tests are more cost-effective and less dangerous than field tests; hence, they are becoming the norm for thoroughly testing self-driving cars in virtual environments. High-quality simulation-based testing requires physically accurate computer simulations, detailed and photo-realistic maps, and systematic approaches for generating tests. Moreover, since creating detailed maps is a manual process, they are expensive, and testers should avoid wasting such valuable resources, for example, by generating irrelevant test cases. To address those issues, we propose SALVO a fully automated approach to identify quantifiably diverse and critical driving scenarios that can be instantiated on existing high-definition maps and implement them in an industrial driving simulator as executable test cases. The evaluation of SALVO in the context of the 2021 IEEE Autonomous Driving AI Test Challenge showed that it could analyze maps of different complexity and identify many critical driving scenarios in minutes. Furthermore, the tests SALVO generated stressed a state-of-art self-driving car software in quantifiably diverse ways and exposed issues with its implementation.","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"118 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116375324","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Quantifying Nondeterminism and Inconsistency in Self-organizing Map Implementations","authors":"Sydur Rahaman, Raina Samuel, Iulian Neamtiu","doi":"10.1109/AITEST52744.2021.00026","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00026","url":null,"abstract":"Self-organizing maps (SOMs) are a popular approach for neural network-based unsupervised learning. However the reliability of self-organizing map implementations has not been investigated. Using internal and external metrics, we define and check two basic SOM properties. First, determinism: a given SOM implementation should produce the same SOM when run repeatedly on the same training dataset. Second, consistency: two SOM implementations should produce similar SOMs when presented with the same training dataset. We check these properties in four popular SOM implementations. We ran our approach on 381 popular datasets used in health, medicine, and other critical domains. We found that implementations violate these basic properties. For example, 375 out of 381 datasets have nondeterministic outcomes; for 51–92% of datasets, toolkits yield significantly different SOM clusterings; and clustering accuracy might be so inconsistent as to vary by a factor of four between toolkits. This undermines SOM reliability, and the reliability of results obtained via SOMs. Our study shines a light on what to expect, in practice, when running actual SOM implementations. Our findings suggest that for critical applications, SOM users should not take reliability for granted; rather, multiple runs and different toolkits should be considered and compared.","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"361 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122779700","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Evolutionary Generation of Test Case for Deep Neural Network Based on Coverage Guidance","authors":"Liming Hu, Shunhui Ji, Qiyin Dai, Pengcheng Zhang","doi":"10.1109/AITEST52744.2021.00014","DOIUrl":"https://doi.org/10.1109/AITEST52744.2021.00014","url":null,"abstract":"In the past few years, Deep Neural Network(DNN) has made great progress. However, it is difficult to guarantee that DNN-based applications can get satisfactory results. Testing is an effective technology to improve the accuracy and robustness of DNN, in which test case generation is an important task. In this paper, we combine the differential evolution algorithm and coverage criterion which is stronger than neuron coverage to generate test cases for DNN. This method uses the prediction loss of DNN and coverage criterion as the fitness function to generate small perturbations at the pixel level between different channels of the image. Then the generated perturbations are added to the image to construct a test case. Theoretically, the test cases generated by this method can not only make the neural network produce misclassification, but also achieve higher coverage for neurons in the DNN, so that defects in DNN can be detected more comprehensively.","PeriodicalId":366884,"journal":{"name":"2021 IEEE International Conference on Artificial Intelligence Testing (AITest)","volume":"31 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116169305","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}