{"title":"Massively Parallel, Highly Efficient, but What About the Test Suite Quality? Applying Mutation Testing to GPU Programs","authors":"Qianqian Zhu, A. Zaidman","doi":"10.1109/icst46399.2020.00030","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00030","url":null,"abstract":"Thanks to rapid advances in programmability and performance, GPUs have been widely applied in High Performance Computing (HPC) and safety-critical domains. As such, quality assurance of GPU applications has gained increasing attention. This brings us to mutation testing, a fault-based testing technique that assesses the test suite quality by systematically introducing small artificial faults. It has been shown to perform well in exposing faults. In this paper, we investigate whether GPU programming can benefit from mutation testing. In addition to conventional mutation operators, we propose nine GPU-specific mutation operators based on the core syntax differences between CPU and GPU programming. We conduct a preliminary study on six CUDA systems. The results show that mutation testing can effectively evaluate the test quality of GPU programs: conventional mutation operators can guide the engineers to write simple direct tests, while GPU-specific mutation operators can lead to more intricate test cases which are better at revealing GPU-specific weaknesses.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"62 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121085472","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Poster: SONAR Testing – Novel Testing Approach Based on Operation Recording and Visualization","authors":"Hiroyuki Kirinuki, Toshiyuki Kurabayashi, Haruto Tanno, Ippei Kumagawa","doi":"10.1109/icst46399.2020.00050","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00050","url":null,"abstract":"Scripted testing and exploratory testing are widely adopted among developers. It is known that exploratory testing can detect bugs more efficiently than scripted testing. However, exploratory testing has the problem that it is difficult to agree with third parties on test results and test quality. In this study, we propose a novel testing approach to facilitate efficient testing and quality assurance. Furthermore, we propose a tool that records the details of testing activity and visualizes them in multiple ways. Our approach realizes both high efficiency like exploratory testing and high auditability like scripted testing. We interviewed 14 developers about our approach to refine it and make it practical. We describe how we reflected the feedback from the interview in our approach.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"26 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132961178","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"STICCER: Fast and Effective Database Test Suite Reduction Through Merging of Similar Test Cases","authors":"A. Alsharif, G. M. Kapfhammer, Phil McMinn","doi":"10.1109/icst46399.2020.00031","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00031","url":null,"abstract":"Since relational databases support many software applications, industry professionals recommend testing both database queries and the underlying database schema that contains complex integrity constraints. These constraints, which include primary and foreign keys, NOT NULL, and arbitrary CHECK constraints, are important because they protect the consistency and coherency of data in the relational database. Since testing integrity constraints is potentially an arduous task, human testers can use new tools to automatically generate test suites that effectively find schema faults. However, these tool-generated test suites often contain many lengthy tests that may both increase the time overhead of regression testing and limit the ability of human testers to understand them. Aiming to reduce the size of automatically generated test suites for database schemas, this paper introduces STICCER, a technique that finds overlaps between test cases, merging database interactions from similar tests and removing others. By systematically discarding and merging redundant tests, STICCER creates a reduced test suite that is guaranteed to have the same coverage as the original one. Using thirty-four relational database schemas, we experimentally compared STICCER to two greedy test suite reduction techniques and a random method. The results show that, compared to the greedy and random methods, STICCER is the most effective at reducing the number of test cases and database interactions while maintaining test effectiveness as measured by the mutation score.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"60 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127232269","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Toward Automated Assessment of User Experience in Extended Reality","authors":"Saba Gholizadeh Ansari","doi":"10.1109/icst46399.2020.00056","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00056","url":null,"abstract":"Designers of extended reality systems need to predict users feedback about designed elements to evaluate their systems. Manual user experience testing can not cover all preferences of users and user-system interactions. To improve and accelerate this process, automated user experience testing is a field of growing interest. Since users’ emotions affect their experience, the automated testing framework should represent users with different emotional states. In this study, we propose an approach to deploy an automated user experience testing framework using BDI test agents which work with a computational model of emotion to regulate their testing behavior.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"57 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122084892","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Poster: Test Case Prioritization Using Error Propagation Probability*","authors":"Jeonghyun Joo, Seunghoon Yoo, Myunghwan Park","doi":"10.1109/icst46399.2020.00047","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00047","url":null,"abstract":"A software test is to execute the program using a test case to examine whether it produces the intended output. For successful regression testing, it is very important to choose a relatively small amount, but productive test cases to maximize testing efficiency. Test case prioritization is a suggested technique for this purpose. This technique arranges the test cases in such a way that higher-order test cases are expected to outperform those on lower-order test cases in fault-finding capability. In this paper, we suggest a new metric for test case prioritization based on the error propagation probability of the test cases. This metric arranges test cases in order by means of the probabilistic fault finding capability of the test cases. Since our metric is based on mathematical probability, it can show statistically consistent and constant results for the fault-finding capability of test cases. The experiment results show that there is a high correlation between the test cases aligned by our metric and their fault-finding capabilities.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"91 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116264934","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A. Bertolino, G. D. Angelis, Breno Miranda, P. Tonella
{"title":"Run Java Applications and Test Them In-Vivo Meantime","authors":"A. Bertolino, G. D. Angelis, Breno Miranda, P. Tonella","doi":"10.1109/icst46399.2020.00061","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00061","url":null,"abstract":"The outcome of test case execution depends on the state of the object under test. While testers can carefully choose meaningful and representative object states for test execution, it is unaffordable to cover the combinatorial space of possible object states exhaustively. An appealing option is to delegate part of the testing activities to the runtime and to execute test cases in the field whenever a new or uncommon state is observed. We have designed and developed Groucho, a framework for in-vivo testing of Java applications. Among the challenges that we faced, the most important ones are isolation of the test session from the user session and minimal performance overhead. Experimental results show that if the activation probability is kept reasonably small (e.g., $10 ^{- {4}}$), the impact of the framework is imperceptible(i.e., either statistically insignificant or with a negligible effect size).","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"17 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130555681","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Anomaly Analyses to Guide Software Testing Activity","authors":"Allan Mori","doi":"10.1109/icst46399.2020.00055","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00055","url":null,"abstract":"Software testing is a quality activity with the purpose of revealing errors not yet discovered. Also, this activity verifies if the software realizes its functionalities conformance with its specification. As software testing activity evolves during software development, a large set of test cases can be generated, compromising the effort during the regression testing, for instance. Regression testing is a type of software testing to confirm that a new program or code change has not adversely affected existing features of the software. In this activity, the tester may not have enough time to run all test cases, needing to decide which test cases are best in terms of effectiveness in reveal faults. In another way, during code refactoring activities, code smells can be identified, which represents unwanted situations that can negatively impact software quality. This project proposes an approach that employs information about code smells to guide software testing activity. Our hypothesis is that the code smells can guide, for instance, the prioritization of test cases, focusing on effort reduction of software testing. I would like to acknowledged my advisor, Simone R. S. Souza for guiding and supporting in this work.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"446 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115929097","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Learning How to Search: Generating Exception-Triggering Tests Through Adaptive Fitness Function Selection","authors":"H. Almulla, Gregory Gay","doi":"10.1109/icst46399.2020.00017","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00017","url":null,"abstract":"Search-based test generation is guided by feedback from one or more fitness functions—scoring functions that judge solution optimality. Choosing informative fitness functions is crucial to meeting the goals of a tester. Unfortunately, many goals—such as forcing the class-under-test to throw exceptions— do not have a known fitness function formulation. We propose that meeting such goals requires treating fitness function identification as a secondary optimization step. An adaptive algorithm that can vary the selection of fitness functions could adjust its selection throughout the generation process to maximize goal attainment, based on the current population of test suites. To test this hypothesis, we have implemented two reinforcement learning algorithms in the EvoSuite framework, and used these algorithms to dynamically set the fitness functions used during generation.We have evaluated our framework, EvoSuiteFIT, on a set of 386 real faults. EvoSuiteFIT discovers and retains more exception-triggering input and produces suites that detect a variety of faults missed by the other techniques. The ability to adjust fitness functions allows EvoSuiteFIT to make strategic choices that efficiently produce more effective test suites.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"68 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121025590","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"NodeRacer: Event Race Detection for Node.js Applications","authors":"A. T. Endo, Anders Møller","doi":"10.1109/icst46399.2020.00022","DOIUrl":"https://doi.org/10.1109/icst46399.2020.00022","url":null,"abstract":"The Node.js platform empowers a huge number of software systems programmed with JavaScript. Node.js employs an asynchronous execution model where event handlers are scheduled nondeterministically, and unexpected races between event handlers often cause malfunctions. Existing techniques for detecting such event races require complex modifications of the Node.js internals, or target only certain kinds of races. This paper presents a new approach, called NODERACER, that detects event races in Node.js applications by selectively postponing events, guided by happens-before relations. The technique is implemented entirely with code instrumentation, without modifications of the Node.js system. Our experimental results give evidence that NODERACER finds event race errors with higher probability than a state-of-the-art fuzzer, and that the use of happens-before relations helps avoiding false positives. Furthermore, we demonstrate that NODERACER produces actionable error reports, and that it can be helpful for detecting test flakiness that is caused by event races.","PeriodicalId":235967,"journal":{"name":"2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)","volume":"12 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128646835","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}