2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)最新文献

{"title":"SPECMATE: Automated Creation of Test Cases from Acceptance Criteria","authors":"Jannik Fischbach, Andreas Vogelsang, Dominik Spies, A. Wehrle, Maximilian Junker, Dietmar Freudenstein","doi":"10.1109/icst46399.2020.00040","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00040","url":null,"abstract":"In the agile domain, test cases are derived from acceptance criteria to verify the expected system behavior. However, the design of test cases is laborious and has to be done manually due to missing tool support. Existing approaches for automatically deriving tests require semi-formal or even formal notations of acceptance criteria, though informal descriptions are mostly employed in practice. In this paper, we make three contributions: (1) a case study of 961 user stories providing an insight into how user stories are formulated and used in practice, (2) an approach for the automatic extraction of test cases from informal acceptance criteria and (3) a study demonstrating the feasibility of our approach in cooperation with our industry partner. In our study, out of 604 manually created test cases, 56 % can be generated automatically and missing negative test cases are added.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"23 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124212203","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"ICST 2020 Breaker Page","authors":"","doi":"10.1109/icst46399.2020.00003","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00003","url":null,"abstract":"","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129459884","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"More Accurate Dynamic Slicing for Better Supporting Software Debugging","authors":"Xiangyu Li, A. Orso","doi":"10.1109/ICST46399.2020.00014","DOIUrl":"https://doi.org/10.1109/ICST46399.2020.00014","url":null,"abstract":"Dynamic slicing and its underlying dynamic dependence analysis have been extensively studied and used as the foundation for numerous automated-debugging techniques. One limitation of dynamic slicing, when used for debugging, is that it only considers program dependences that are actually observed during the execution(s) of interest. Some faults, however, involve potential, rather than actual dependences—dependences that would be observed if the correct program was executed but are missing when the faulty program is executed. In particular, traditional dynamic slicing may fail to locate faults that involve assignments that should have occurred in a correct execution and did not occur in the failing execution being debugged. Relevant slicing techniques partially address this problem by identifying missing assignments due to incorrect control-flow. However, they do not consider the case of assignments that do occur but modify the wrong memory location (e.g., the wrong element of an array). Debugging techniques based on existing dynamic slicing approaches may therefore miss faults in the presence of this kind of incorrect assignments. To address this problem, we introduce the concept of potential memory-address dependence (PMD). Intuitively, PMDs represent the dependence relationship between an instruction s that affects the computation of a memory-address ma (e.g., by defining an array index or a pointer offset) and memory read instructions that are not observed to be dependent on s but could be affected by s (i.e., access the memory at ma) in a counterfactual execution of s. We also present a technique that computes PMDs and represents them on standard dynamic dependence graphs. To assess the effectiveness of our technique for debugging, we implemented PMD-Slicer, a dynamic slicer that accounts for PMDs, and performed an empirical evaluation on a benchmark of 364 real faults and 880 fault-revealing test cases. Our results are promising, in that almost 10% of the failing tests contained cases in which PMD-Slicer generated slices that included the corresponding fault, while a traditional dynamic slicer did not. Furthermore, considering PMDs only moderately increased slice sizes.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"125 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121204733","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Poster: Is Euclidean Distance the best Distance Measurement for Adaptive Random Testing?","authors":"Rubing Huang, Chenhui Cui, Weifeng Sun, D. Towey","doi":"10.1109/icst46399.2020.00049","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00049","url":null,"abstract":"Adaptive random testing (ART) aims at enhancing the testing effectiveness of random testing (RT) by more evenly spreading test cases over the input domain. Many ART methods have been proposed, based on various, different notions. For example, distance-based ART (DART) makes use of the concept of distance to implement ART, attempting to generate new test cases that are far away from previously executed ones. The Euclidean distance has been a popular choice of distance metric, used in DART to evaluate the differences between test cases. However, is the Euclidean distance the most suitable choice for DART? To answer this question, we conducted a series of simulations to investigate the impact that the Euclidean distance, and its many variations, has on the testing effectiveness of DART. The results show that when the dimensionality of the input domain is low, the Euclidean distance may indeed be a good choice. However, when the dimensionality is high, it appears to be less suitable.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"99 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127103418","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Detecting Higher-Order Merge Conflicts in Large Software Projects","authors":"Thorsten Wuensche, A. Andrzejak, Sascha Schwedes","doi":"10.1109/icst46399.2020.00043","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00043","url":null,"abstract":"Merge conflicts can occur when multiple developers work concurrently on the same source code corpus. Diverging textual changes (in the same lines of code) are typically easy to resolve with help of common tools like Git. More challenging are higher-order merge conflicts. They arise as the result of unintended interactions between changes in different parts of the source code. Higher-order merge conflicts can be caused by a combination of changes, and so even thorough testing of the individual development branches might not be able to identify them. We suggest an approach based on static analysis and a prototypical tool to detect potential higher-order merge conflicts. Our method identifies potentially dangerous dependencies between changed code fragments in a call graph. An evaluation on SAP HANA, a very large industrial product in C++, shows that the approach is able to identify 62% of higher-order merge conflicts causing build failures over 22 months project development time. The same prototype finds no instance of higher-order merge conflicts causing test failures in SAP HANA during a two month development period. In summary, our method scales well and can identify higher-order merge conflicts which escape traditional testing.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129012877","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"ICST 2020 Commentary","authors":"","doi":"10.1109/icst46399.2020.00001","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00001","url":null,"abstract":"","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130476245","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"An Empirical Evaluation of Mutation Operators for Deep Learning Systems","authors":"Gunel Jahangirova, P. Tonella","doi":"10.1109/icst46399.2020.00018","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00018","url":null,"abstract":"Deep Learning (DL) is increasingly adopted to solve complex tasks such as image recognition or autonomous driving. Companies are considering the inclusion of DL components in production systems, but one of their main concerns is how to assess the quality of such systems. Mutation testing is a technique to inject artificial faults into a system, under the assumption that the capability to expose (kilt) such artificial faults translates into the capability to expose also real faults. Researchers have proposed approaches and tools (e.g., Deep-Mutation and MuNN) that make mutation testing applicable to deep learning systems. However, existing definitions of mutation killing, based on accuracy drop, do not take into account the stochastic nature of the training process (accuracy may drop even when re-training the un-mutated system). Moreover, the same mutation operator might be effective or might be trivial/impossible to kill, depending on its hyper-parameter configuration. We conducted an empirical evaluation of existing operators, showing that mutation killing requires a stochastic definition and identifying the subset of effective mutation operators together with the associated most effective configurations.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"154 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128159482","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Automated verification of compliance of non-functional requirements on mobile applications through metamorphic testing","authors":"M. C. Júnior","doi":"10.1109/ICST46399.2020.00053","DOIUrl":"https://doi.org/10.1109/ICST46399.2020.00053","url":null,"abstract":"The growing use of mobile applications in almost every aspect of modern life, from banking to office systems, is making it is quality increasingly necessary. Thus, software testing activities have been increasingly in demand in this domain. However, mobile application testing imposes several new challenges and several peculiarities, such as limited power, interface adaptation, and privacy of user data. Most of these peculiarities are related to Non-Functional Requirements (NFRs) and conventional software testing approaches are ineffective. Based on this, the Metamorphic Testing (MT) is an approach that has been applied in many domains as a strategy for generating new test cases and an alternative to alleviate the oracle problem. Through the generation of Metamorphic Relations (MRs), generated from characteristics of the System Under Testing (SUT), it is possible to evaluate the presence of flaws. Recently, studies have been conducted to evaluate the applicability of MT to reveal performance flaws. The goal of this Ph.D. project proposal is to propose an approach to automate the verification of compliance of NFRs on mobile applications through MT. Also, we intend to develop a tool for applying the approach, and a set of criteria to improve the process of verifying NFRs through MT. Finally, we expect to evaluate the effectiveness of the approach on real mobile applications.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"8 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128320341","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Prioritizing Runtime Verification Violations","authors":"Breno Miranda, Igor Lima, Owolabi Legunsen, Marcelo d’Amorim","doi":"10.1109/ICST46399.2020.00038","DOIUrl":"https://doi.org/10.1109/ICST46399.2020.00038","url":null,"abstract":"Runtime Verification (RV) can help find software bugs by monitoring formally specified properties during testing. A key problem when using RV during testing is how to reduce the manual inspection effort for checking whether property violations are true bugs. To date, there was no automated approach for determining the likelihood that property violations were true bugs to reduce tedious and time-consuming manual inspection.We present RVPRIO, the first automated approach for prioritizing RV violations in order of likelihood of being true bugs. RVPRIO uses machine learning classifiers to prioritize violations. For training, we used a labeled dataset of 1,170 violations from 110 projects. On that dataset, (1) RVPRIO reached 90% of the effectiveness of a theoretically optimal prioritizer that ranks all true bugs at the top of the ranked list, and (2) 88.1% of true bugs were in the top 25% of RVPRIO-ranked violations; 32.7% of true bugs were in the top 10%. RVPRIO was also effective when we applied it to new unlabeled violations, from which we found previously unknown bugs—29 bugs in 7 projects and two bugs in two properties. Our dataset is publicly available online.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"23 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133321171","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"An Empirical Analysis of Blind Tests","authors":"Kesina Baral, Jeff Offutt","doi":"10.1109/icst46399.2020.00034","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00034","url":null,"abstract":"Modern software engineers automate as many tests as possible. Test automation allows tests to be run hundreds or thousands of times: hourly, daily, and sometimes continuously. This saves time and money, ensures reproducibility, and ultimately leads to software that is better and cheaper. Automated tests must include code to check that the output of the program on the test matches expected behavior. This code is called the test oracle and is typically implemented in assertions that flag the test as passing if the assertion evaluates to true and failing if not. Since automated tests require programming, many problems can occur. Some lead to false positives, where incorrect behavior is marked as correct, and others to false negatives, where correct behavior is marked as incorrect. This paper identifies and studies a common problem where test assertions are written incorrectly, leading to incorrect behavior that is not recognized. We call these tests blind because the test does not see the incorrect behavior. Blind tests cause false positives, essentially wasting the tests. This paper presents results from several human-based studies to assess the frequency of blind tests with different software and different populations of users. In our studies, the percent of blind tests ranged from a low of 39% to a high of 95%.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"39 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122356810","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}