{"title":"Automated GUI-based Software-Testing Using Deep Neuroevolution","authors":"Daniel Zimmermann","doi":"10.1109/ICST53961.2022.00060","DOIUrl":"https://doi.org/10.1109/ICST53961.2022.00060","url":null,"abstract":"Testing software applications is an important part of the development process. It improves the quality by reducing possible misbehavior. A common testing method is manual testing, which involves human interaction with the graphical user interface (GUI). Here, testers use their experience and intuition to uncover possible errors, but this is very time-consuming and expensive. In contrast, when using test scripts, a large part of the application under test (AUT) can be examined in a short time by running a high number of test cases, but hidden errors often remain undetected. In addition, such approaches are inflexible in their execution, have a high maintenance effort with a new release, and require a very good system understanding and programming knowledge to create the tests.","PeriodicalId":188674,"journal":{"name":"2022 IEEE Conference on Software Testing, Verification and Validation (ICST)","volume":"55 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121145770","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Automated Repair of Responsive Web Page Layouts","authors":"Ibrahim Althomali, G. M. Kapfhammer, Phil McMinn","doi":"10.1109/ICST53961.2022.00025","DOIUrl":"https://doi.org/10.1109/ICST53961.2022.00025","url":null,"abstract":"Responsive Web Design (RWD) is a strategy that allows developers to create webpages that adjust their layout according to available screen size. Since modern web applications must format correctly on the small displays of mobile devices up to the large displays on desktop computers, and given this dramatic difference in screen space, Responsive Layout Failures (RLFs) - visual discrepancies that are only apparent at certain screen sizes - can easily creep into live production webpages. These can include, for example, HTML elements protruding off the edge of the page or into one another as layout space becomes scarce. This leaves webpages looking unprofessional at best and non-functional at worst. This paper presents a technique for repairing RLFs, implemented into a tool called Layout Dr. After detecting an RLF, Layout Dr harvests layouts from the page's responsive design that are closest to the point of failure, but where the RLF does not occur. It then transforms these layouts so that they can be transplanted over the failure, effectively “hiding” the original RLF from the end user. We evaluated Layout Dr on 19 subjects, containing 55 RLFs in total. Layout Dr could find a suitable fix for each of them. When we conducted a human study of the repairs, 92% of the participants preferred the repaired version of the page compared to the original containing the RLF.","PeriodicalId":188674,"journal":{"name":"2022 IEEE Conference on Software Testing, Verification and Validation (ICST)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133345211","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Owain Parry, G. M. Kapfhammer, Michael C Hilton, Phil McMinn
{"title":"Evaluating Features for Machine Learning Detection of Order- and Non-Order-Dependent Flaky Tests","authors":"Owain Parry, G. M. Kapfhammer, Michael C Hilton, Phil McMinn","doi":"10.1109/ICST53961.2022.00021","DOIUrl":"https://doi.org/10.1109/ICST53961.2022.00021","url":null,"abstract":"Flaky tests are test cases that can pass or fail without code changes. They often waste the time of software developers and obstruct the use of continuous integration. Previous work has presented several automated techniques for detecting flaky tests, though many involve repeated test executions and a lot of source code instrumentation and thus may be both intrusive and expensive. While this motivates researchers to evaluate machine learning models for detecting flaky tests, prior work on the features used to encode a test case is limited. Without further study of this topic, machine learning models cannot perform to their full potential in this domain. Previous studies also exclude a specific, yet prevalent and problematic, category of flaky tests: order-dependent (OD) flaky tests. This means that prior research only addresses part of the challenge of detecting flaky tests with machine learning. Closing this knowledge gap, this paper presents a new feature set for encoding tests, called Flake16. Using 54 distinct pipelines of data preprocessing, data balancing, and machine learning models for detecting both non-order-dependent (NOD) and OD flaky tests, this paper compares Flake16 to another well-established feature set. To assess the new feature set's effectiveness, this paper's experiments use the test suites of 26 Python projects, consisting of over 67,000 tests. Along with identifying the most impactful metrics for using machine learning to detect both types of flaky test, the empirical study shows how Flake16 is better than prior work, including (1) a 13% increase in overall F1 score when detecting NOD flaky tests and (2) a 17% increase in overall F1 score when detecting OD flaky tests.","PeriodicalId":188674,"journal":{"name":"2022 IEEE Conference on Software Testing, Verification and Validation (ICST)","volume":"214 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128169246","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"GUI Test Transfer from Web to Android","authors":"Jun-Wei Lin, S. Malek","doi":"10.1109/ICST53961.2022.00011","DOIUrl":"https://doi.org/10.1109/ICST53961.2022.00011","url":null,"abstract":"GUI testing is important for examining the end-to-end workflows and usability of GUI-based software. To reduce the manual effort of writing GUI tests, recent research has explored the potential of automatically reusing GUI tests by transferring them across similar applications. However, what is missing from the prior work is that such transfer may be required for apps available on different platforms. In particular, both web and Android are dominant platforms on which many organizations provide their software services. At the state-of-the-practice, even if the web and Android versions of an app provisioned by an organization substantially share the functionality, the developers have to manually write separate sets of tests for each version. This paper proposes TransDroid, an automated tool that transfers GUI tests from a web app to its Android counterpart. Evaluation of TransDroid on real-world web and Android apps corroborates its effectiveness by achieving 77% success rate among the attempted transfers, along with 82% precision and 99% recall in the mapping of the GUI events and oracles.","PeriodicalId":188674,"journal":{"name":"2022 IEEE Conference on Software Testing, Verification and Validation (ICST)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130023394","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Symbolic Verification of Message Signatures in MPI","authors":"Hengbiao Yu, Banghu Yin, Xin Yi","doi":"10.1109/ICST53961.2022.00029","DOIUrl":"https://doi.org/10.1109/ICST53961.2022.00029","url":null,"abstract":"The Message Passing Interface (MPI) is the standard paradigm of programming in high performance computing. However, the inherent complexity and the large size of MPI standard make it difficult for programmers to use the MPI APIs correctly. This paper focuses on the mismatch errors of message signatures. Considering that MPI errors may occur during some intricate, low probability interleavings under specific inputs, we adopt symbolic verification to verify the correct match of message signatures. Specifically, we propose a precise method for modeling the match of message signatures of an execution path in terms of communicating sequential processes. To improve the scalability, we give a partial order reduction based optimization to reduce the complexity of path-level communication models. We have implemented our method as a prototype tool and evaluated it on the typical correctness benchmark MPI-Corbench and 8 real-world open source MPI programs, totaling 37K lines of code. The experimental results demonstrate the effectiveness and scalability of our method.","PeriodicalId":188674,"journal":{"name":"2022 IEEE Conference on Software Testing, Verification and Validation (ICST)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129288911","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Andrei Lascu, A. Donaldson, T. Grosser, T. Hoefler
{"title":"Metamorphic Fuzzing of C++ Libraries","authors":"Andrei Lascu, A. Donaldson, T. Grosser, T. Hoefler","doi":"10.1109/ICST53961.2022.00015","DOIUrl":"https://doi.org/10.1109/ICST53961.2022.00015","url":null,"abstract":"We present a method for automated metamorphic fuzzing of software libraries, implemented as an open-source tool, MF++, targeting C++ libraries. Our approach works by automatically synthesising equivalent sequences of calls to a library's API based on a user-provided specification, in a randomized fashion. Equivalent call sequences are then tested using randomized inputs, and result mismatches reveal bugs in the library implementation. This is an instance of metamorphic testing: it avoids the oracle problem because we do not need to know the expected results of a set of equivalent call sequences, only that their results should match. Automated test case reduction can then be used to find minimized equivalent call sequences that trigger mismatches, as an aid to debugging. We evaluate MF++ with respect to four SMT solving libraries and two Presburger arithmetic libraries, leading to the discovery of 21 bugs. We have also successfully used MF++ and its test case reduction facilities to automatically generate small test cases that exercise source code not covered by the regression test suites of various libraries under test. Unlike most test case generation techniques, the tests we synthesise are equipped with an oracle by construction: the equivalence-based oracle offered by our metamorphic approach. We have submitted patches contributing new coverage-enhancing test cases to the isl, Yices2 and Z3 projects. The developers of these projects have accepted 21 tests based on our patches so far.","PeriodicalId":188674,"journal":{"name":"2022 IEEE Conference on Software Testing, Verification and Validation (ICST)","volume":"159 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116155347","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Locating Code Omission Error due to Incorrect Polymorphic Method Call","authors":"S. Dutta, Debarshi Kumar Sanyal","doi":"10.1109/ICST53961.2022.00057","DOIUrl":"https://doi.org/10.1109/ICST53961.2022.00057","url":null,"abstract":"Dynamic program slicing methods are widely used for debugging because many statements can be ignored in the process of localizing a bug. A dynamic program slice for a variable contains only those statements that influenced this variable. One limitation of dynamic slicing-based techniques is that they cannot capture execution omission errors, which may cause the execution of certain critical statements in a program to be omitted and thus result in failures. In this paper, we propose a solution to locate execution omission errors by using dynamic slices for a variable in the presence of methods that are virtual in Java programs. Note that any method which is not a static, private or final method can be considered a virtual method in a Java program. Due to the assignment of objects of wrong derived classes to base class reference, some different versions of polymorphic methods can be called which may cause failure. We designed a slicing method called polymorphic relevant slice which can be used to force the execution of the omitted code by initializing objects of all possible classes derived from the same base class with the available data, assigning them to base class reference one-by-one and switch execution for all alternative virtual functions to check if any of them meets specification. We have used a system dependence graph to identify all the derived class objects which could have been assigned to the base class reference.","PeriodicalId":188674,"journal":{"name":"2022 IEEE Conference on Software Testing, Verification and Validation (ICST)","volume":"22 18","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131614370","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"RiverGame - a game testing tool using artificial intelligence","authors":"C. Paduraru, Miruna Paduraru, Alin Stefanescu","doi":"10.1109/ICST53961.2022.00048","DOIUrl":"https://doi.org/10.1109/ICST53961.2022.00048","url":null,"abstract":"As is the case with any very complex and interactive software, many video games are released with various minor or major issues that can potentially affect the user experience, cause security issues for players, or exploit the companies that deliver the products. To test their games, companies invest important resources in quality assurance personnel who usually perform the testing mostly manually. The main goal of our work is to automate various parts of the testing process that involve human users (testers) and thus to reduce costs and run more tests in less time. The secondary goal is to provide mechanisms to make test specification writing easier and more efficient. We focus on solving initial real-world problems that have emerged from several discussions with industry partners. In this paper, we present RiverGame, a tool that allows game developers to automatically test their products from different points of view: the rendered output, the sound played by the game, the animation and movement of the entities, the performance and various statistical analyses. We also address the problem of input priorities, scheduling, and directing the testing effort towards custom and dynamic directions. At the core of our methods, we use state-of-the-art artificial intelligence methods for analysis and a behavior-driven development (BDD) methodology for test specifications. Our technical solution is open-source, independent of game engine, platform, and programming language.","PeriodicalId":188674,"journal":{"name":"2022 IEEE Conference on Software Testing, Verification and Validation (ICST)","volume":"172 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133751914","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Rachel Tzoref, S. Sinha, Antonio Abu Nassar, V. Goldin, Haim Kermany
{"title":"TackleTest: A Tool for Amplifying Test Generation via Type-Based Combinatorial Coverage","authors":"Rachel Tzoref, S. Sinha, Antonio Abu Nassar, V. Goldin, Haim Kermany","doi":"10.1109/ICST53961.2022.00050","DOIUrl":"https://doi.org/10.1109/ICST53961.2022.00050","url":null,"abstract":"We present TackleTest, an open-source tool for automatic generation of unit-level test cases for Java applications. TackleTest builds on top of two well-known test-generation tools, EvoSuite and Randoop, by adding a new combinatorial-testing-based approach for computing coverage goals that comprehensively exercises different parameter type combinations of the methods under test, at configurable interaction levels. We describe the tool architecture, the main tool components, and the combinatorial type-based testing technique. TackleTest was developed in the context of application modernization at IBM, but it is also applicable as a general-purpose test-generation tool. We have evaluated TackleTest on several IBM-internal enterprise applications as well as on a subset of the SF110 benchmark, and share our findings and lessons learned. Overall, TackleTest implements a new and complementary way of computing coverage goals for unit testing via a novel white-box application of combinatorial testing.","PeriodicalId":188674,"journal":{"name":"2022 IEEE Conference on Software Testing, Verification and Validation (ICST)","volume":"66 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114754611","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"To Seed or Not to Seed? An Empirical Analysis of Usage of Seeds for Testing in Machine Learning Projects","authors":"Saikat Dutta, Anshul Arunachalam, Sasa Misailovic","doi":"10.1109/ICST53961.2022.00026","DOIUrl":"https://doi.org/10.1109/ICST53961.2022.00026","url":null,"abstract":"Many Machine Learning (ML) algorithms are in-herently random in nature - executing them using the same inputs may lead to slightly different results across different runs. Such randomness makes it challenging for developers to write tests for their implementations of ML algorithms. A natural consequence of randomness is test flakiness - tests both pass and fail non-deterministically for same version of code. Developers often choose to alleviate test flakiness in ML projects by setting seeds in the random number generators used by the code under test. However, this approach commonly serves as a “workaround” rather than an actual solution. Instead, it may be possible to mitigate flakiness and alleviate the negative effects of setting seeds using alternative approaches. To understand the role of seeds and the feasibility of alternative solutions, we conduct the first large-scale empirical study of the usage of seeds and its implications on testing on a corpus of 114 Machine Learning projects. We identify 461 tests in these projects that fail without seeds and study their nature and root causes. We try to minimize the flakiness of a subset of 42 identified tests using alternative strategies such as tuning algorithm hyper-parameters and adjusting assertion bounds and send them to developers. So far, developers have accepted our fixes for 26 tests. We further manually analyze a subset of 56 tests and study various characteristics such as the nature of test oracles and how the seed settings evolve over time. Finally, we provide a general set of recommendations for both researchers and developers in the context of setting seeds in tests.","PeriodicalId":188674,"journal":{"name":"2022 IEEE Conference on Software Testing, Verification and Validation (ICST)","volume":"202 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116171529","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}