2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)最新文献

{"title":"RESTTESTGEN: Automated Black-Box Testing of RESTful APIs","authors":"Emanuele Viglianisi, Michael Dallago, M. Ceccato","doi":"10.1109/icst46399.2020.00024","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00024","url":null,"abstract":"RESTful APIs (or REST APIs for short) represent a mainstream approach to design and develop Web APIs using the REpresentational State Transfer architectural style. When their source code is not (or just partially) available or the analysis across many dynamically allocated distributed components (typical of a micro-services architecture) poses obstacles to white-box testing, black-box testing becomes a viable option. Black-box testing, in fact, only assumes access to the system under test with a specific interface. This paper presents RESTTESTGEN, a novel approach to automatically generate test cases for REST APIs, based on their interface definition (in Swagger). Input values and requests are generated for each operation of the API under test, with the twofold objective of testing nominal execution scenarios and of testing error scenarios. Two distinct oracles are deployed to detect when test cases reveal implementation defects. Our empirical investigation shows that this approach is effective in revealing actual faults on 87 real-world REST APIs.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"11 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115380469","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Poster: Performance Testing Driven by Reinforcement Learning","authors":"M. H. Moghadam, Mehrdad Saadatmand, Markus Borg, M. Bohlin, B. Lisper","doi":"10.1109/icst46399.2020.00048","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00048","url":null,"abstract":"Performance testing remains a challenge, particularly for complex systems. Different application-, platform- and workload-based factors can influence the performance of software under test. Common approaches for generating platform- and workload-based test conditions are often based on system model or source code analysis, real usage modeling and use-case based design techniques. Nonetheless, creating a detailed performance model is often difficult, and also those artifacts might not be always available during the testing. On the other hand, test automation solutions such as automated test case generation can enable effort and cost reduction with the potential to improve the intended test criteria coverage. Furthermore, if the optimal way (policy) to generate test cases can be learnt by testing system, then the learnt policy can be reused in further testing situations such as testing variants, evolved versions of software, and different testing scenarios. This capability can lead to additional cost and computation time saving in the testing process. In this research, we present an autonomous performance testing framework which uses a model-free reinforcement learning augmented by fuzzy logic and self-adaptive strategies. It is able to learn the optimal policy to generate platform- and workload-based test conditions which result in meeting the intended testing objective without access to system model and source code. The use of fuzzy logic and self-adaptive strategy helps to tackle the issue of uncertainty and improve the accuracy and adaptivity of the proposed learning. Our evaluation experiments show that the proposed autonomous performance testing framework is able to generate the test conditions efficiently and in a way adaptive to varying testing situations.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"83 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125315332","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"MiMIs: Simple, Efficient, and Fast Bounded-Exhaustive Test Case Generators","authors":"Kyle Dewey, Shant Hairapetian, M. Gavrilov","doi":"10.1109/icst46399.2020.00016","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00016","url":null,"abstract":"Bounded-exhaustive test case generators are important bug-finding tools, epitomized with tools like Korat, UDITA, and SciFe. All such tools are practically limited by scale; a given testing problem may be too large to generate all tests within reasonable time, or prohibitive amounts of memory may be required for this generation. The tools themselves may have lengthy implementations which hide their own bugs, or they may require users to familiarize themselves with many operations. In this paper, we strive to push the boundaries of what test case generators can scale to, while preserving a simple implementation and user interface. To this end, we introduce a novel programming abstraction: Memoized Monadic Iterators (MiMIs). MiMIs combine the disparate ideas of ASTGen’s iterators, LogicT’s additive monad interface, and SciFe’s memoization into a single coherent programming abstraction. Our evaluation shows that MiMIs are typically faster than the state of the art with orders of magnitude less memory used, and that MiMIs overall scale better to big testing problems than any competitor. Thanks to their fundamental design, MiMIs are implemented in under 200 LOC (compared to thousands with most competitors), and require users to familiarize themselves with collectively only 7 operations and types (less than half as many as the state of the art), all without sacrificing generator conciseness.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"303 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122307292","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Well-informed Test Case Generation and Crash Reproduction","authors":"P. Derakhshanfar","doi":"10.1109/icst46399.2020.00054","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00054","url":null,"abstract":"Search-based test data generation approaches have come a long way over the past few years, but these approaches still have some limitations when it comes to exercising specific behavior for triggering particular kinds of faults (e.g., crashes or specific types of integration between classes/modules). In this thesis, we are investigating new fitness functions and evolutionary-based algorithms and techniques to tackle these limitations. We have defined multiple novel approaches for crash reproduction and class integration testing. Currently, we are still working on improving both crash reproduction and class integration testing.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130663743","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"EC.LANG – A Language for Specifying Response Time Analyses of Event Chains","authors":"Max J. Friese, J. Traub, Dirk Nowotka","doi":"10.1109/icst46399.2020.00042","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00042","url":null,"abstract":"Modern cyber-physical systems pose great challenges for system engineers to keep track of the system’s behavior when it comes to functions distributed all over the system. To check whether response time constraints are met, measurement data from different development stages is analyzed to track down the worst-case behavior observed.Several complex, signal dependencies have to be examined over long time periods. Therefore, computer aided approaches to support this task are strongly demanded. In this paper, we present EC.LANG, a formal language designed to specify evaluations over measurement data. It is particularly fitted to model event chains representing the data flow of system functions. To validate event chains against timing requirements, we implemented a compiler and an evaluation engine based on EC.LANG.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114065403","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Family of Experiments to Assess the Impact of Page Object Pattern in Web Test Suite Development","authors":"Maurizio Leotta, Matteo Biagiola, F. Ricca, M. Ceccato, P. Tonella","doi":"10.1109/ICST46399.2020.00035","DOIUrl":"https://doi.org/10.1109/ICST46399.2020.00035","url":null,"abstract":"Automated web testing is an appealing option, especially when continuous testing practices are adopted. However, web test cases are known to be fragile and to break easily when a web application evolves. The Page Object (PO) design pattern addresses such problem by providing a layer of indirection that decouples test cases from the internals of the web page, where web page elements are located and triggered by the web tests. However, PO development could potentially introduce an additional burden to the already strictly constrained testing activities. This paper reports an empirical investigation of costs and benefits due to the introduction of the PO pattern in web test suite development. In particular, we conducted a family of controlled experiments in which test cases were developed with and without the PO pattern. While the benefits of POs did not compensate for the extra development effort they require in the limited experimental setting of our study, results indicate that when the test suite to be developed is at least 10× larger, test development becomes more efficient with than without POs.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"108 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124145820","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Generating Avoidable Collision Scenarios for Testing Autonomous Driving Systems","authors":"Alessandro Calò, Paolo Arcaini, Shaukat Ali, Florian Hauer, F. Ishikawa","doi":"10.1109/icst46399.2020.00045","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00045","url":null,"abstract":"Automated and autonomous driving systems (ADS) are a transformational technology in the mobility sector. Current practice for testing ADS uses virtual tests in computer simulations; search-based approaches are used to find particularly dangerous situations, possibly collisions. However, when a collision is found, it is not always easy to automatically assess whether the ADS should have been able to avoid it, without relying on offline analyses by domain experts. In this paper, we propose a definition of avoidable collision that does not rely on any domain knowledge, but only on the fact that it is possible to reconFigure the ADS (in our case, the path planner component provided by our industry partner) in a way that the collision is avoided. Based on this definition, we propose two search-based approaches for finding avoidable collisions. The first one (named sequential approach), based on current industrial practice, first searches for a collision, and then searches for an alternative configuration of the ADS which avoids it. The second one (named combined approach), instead, searches at the same time for the collision and for the alternative configuration which avoids it. Experiments show that the combined approach finds more avoidable collisions, even when the sequential approach doesn’t find any; indeed, the sequential approach, in the first search, may find too severe collisions for which there is no alternative configuration that can avoid them.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"10 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128429603","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Organizing partners","authors":"","doi":"10.1109/icst46399.2020.00008","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00008","url":null,"abstract":"","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"6 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131112328","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Prioritizing Runtime Verification Violations","authors":"Breno Miranda, Igor Lima, Owolabi Legunsen, Marcelo d’Amorim","doi":"10.1109/ICST46399.2020.00038","DOIUrl":"https://doi.org/10.1109/ICST46399.2020.00038","url":null,"abstract":"Runtime Verification (RV) can help find software bugs by monitoring formally specified properties during testing. A key problem when using RV during testing is how to reduce the manual inspection effort for checking whether property violations are true bugs. To date, there was no automated approach for determining the likelihood that property violations were true bugs to reduce tedious and time-consuming manual inspection.We present RVPRIO, the first automated approach for prioritizing RV violations in order of likelihood of being true bugs. RVPRIO uses machine learning classifiers to prioritize violations. For training, we used a labeled dataset of 1,170 violations from 110 projects. On that dataset, (1) RVPRIO reached 90% of the effectiveness of a theoretically optimal prioritizer that ranks all true bugs at the top of the ranked list, and (2) 88.1% of true bugs were in the top 25% of RVPRIO-ranked violations; 32.7% of true bugs were in the top 10%. RVPRIO was also effective when we applied it to new unlabeled violations, from which we found previously unknown bugs—29 bugs in 7 projects and two bugs in two properties. Our dataset is publicly available online.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"23 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133321171","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"More Accurate Dynamic Slicing for Better Supporting Software Debugging","authors":"Xiangyu Li, A. Orso","doi":"10.1109/ICST46399.2020.00014","DOIUrl":"https://doi.org/10.1109/ICST46399.2020.00014","url":null,"abstract":"Dynamic slicing and its underlying dynamic dependence analysis have been extensively studied and used as the foundation for numerous automated-debugging techniques. One limitation of dynamic slicing, when used for debugging, is that it only considers program dependences that are actually observed during the execution(s) of interest. Some faults, however, involve potential, rather than actual dependences—dependences that would be observed if the correct program was executed but are missing when the faulty program is executed. In particular, traditional dynamic slicing may fail to locate faults that involve assignments that should have occurred in a correct execution and did not occur in the failing execution being debugged. Relevant slicing techniques partially address this problem by identifying missing assignments due to incorrect control-flow. However, they do not consider the case of assignments that do occur but modify the wrong memory location (e.g., the wrong element of an array). Debugging techniques based on existing dynamic slicing approaches may therefore miss faults in the presence of this kind of incorrect assignments. To address this problem, we introduce the concept of potential memory-address dependence (PMD). Intuitively, PMDs represent the dependence relationship between an instruction s that affects the computation of a memory-address ma (e.g., by defining an array index or a pointer offset) and memory read instructions that are not observed to be dependent on s but could be affected by s (i.e., access the memory at ma) in a counterfactual execution of s. We also present a technique that computes PMDs and represents them on standard dynamic dependence graphs. To assess the effectiveness of our technique for debugging, we implemented PMD-Slicer, a dynamic slicer that accounts for PMDs, and performed an empirical evaluation on a benchmark of 364 real faults and 880 fault-revealing test cases. Our results are promising, in that almost 10% of the failing tests contained cases in which PMD-Slicer generated slices that included the corresponding fault, while a traditional dynamic slicer did not. Furthermore, considering PMDs only moderately increased slice sizes.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"125 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121204733","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}