{"title":"Impact of Static and Dynamic Coverage on Test-Case Prioritization: An Empirical Study","authors":"Jianyi Zhou, Dan Hao","doi":"10.1109/ICSTW.2017.74","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.74","url":null,"abstract":"Most of existing research in Test-Case Prioritization uses coverage information as the input during the process of prioritization and these coverage can be classfied into two categories: static coverage and dynamic coverage. As these coverage information are collected in different ways, they have different influence on test-case prioritization. In this work, we present the first empirical study comparing the impact of static coverage and dynamic coverage with five typical techniques at different test-case granularities (e.g., test-method and test-class level) and different coverage criteria (e.g., method and statement coverage). This study is performed on 15 real-world Java projects (using 163 versions) and we find that the dynamic coverage performs better than static coverage in terms of the results of test-case prioritization.","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"124 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114082860","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Defining the Phrase \"Software Test Architecture\" Emerging Idea","authors":"J. Hagar","doi":"10.1109/ICSTW.2017.58","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.58","url":null,"abstract":"This paper examines if the phrase \"software test architecture\" is universally defined and understood. While there may be some agreement on the phrase within select software communities, there is a lack of world standards on the topic. A lack of common definition and practice should be addressed with the maturing of industry segments. This paper considers the lack of a current world industrial definition of \"software test architecture\" and then provides some recommendations on emerging future work to assist in the definition efforts.","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"80 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133859512","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"A Model for T-Way Fault Profile Evolution during Testing","authors":"D. R. Kuhn, R. Kacker, Yu Lei","doi":"10.1109/ICSTW.2017.35","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.35","url":null,"abstract":"Empirical studies have shown that most software interaction faults involve one or two variables interacting, with progressively fewer triggered by three to six variables interacting. This paper introduces a model for the origin of this distribution. We start with two empirically reasonable assumptions regarding the distribution of branch conditions in code and the proportion of t-way combinations seen in testing, and show that the model closely reproduces empirical data on t-way fault distributions. Although the model was developed to explain the distribution of faults by t-way interaction strength, it is shown to reproduce the basic exponential reliability model as a special case at each level of interaction, t. The paper evaluates model predictions against empirical data, and discusses implications for detection and removal of interaction faults.","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"8 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130151896","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"General Message from the IWCT Workshop Chairs","authors":"D. Simos, Rachel Tzoref","doi":"10.1109/ICSTW.2017.33","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.33","url":null,"abstract":"None","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"25 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121196478","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Test Case Generation and Prioritization: A Process-Mining Approach","authors":"Andrea Janes","doi":"10.1109/ICSTW.2017.11","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.11","url":null,"abstract":"Test cases are an essential tool in software quality assurance: they ensure that code behaves as specified in the requirement. However, writing test cases does not have only benefits, it comes with a cost: the programmer has to formulate the test cases and maintain them when the tested source code changes. Particularly for start-ups or small enterprises such costs become prohibitive, which often prefer to invest their time into the development of new functionalities instead of testing. This paper explores the use of process-mining as an approach to create a model of how users interact with a system to a) generate test cases and b) prioritize them. Using process-mining, it is possible to mine from the user behaviour which parts of the system are the most used, in which order they are executed, generate test cases repeating user input, and prioritizing test cases.","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"166 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115846575","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Testing Cache Side-Channel Leakage","authors":"Tiyash Basu, Sudipta Chattopadhyay","doi":"10.1109/ICSTW.2017.16","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.16","url":null,"abstract":"Cache timing attacks retrieve secret information (e.g. a secret key) about a program by analyzing the cache behaviour in program executions. It is, therefore, crucial to understand whether a program is vulnerable to cache timing attacks. But how can we test a program to discover its vulnerability against cache timing attacks? In this paper, we propose, design and evaluate a test generation methodology that systematically discovers the cache side-channel leakage of arbitrary software binaries. At the core of our test generation is a method that systematically explores the program input space and it adapts based on the observed cache performance in the executed tests. We have implemented our test generator and evaluated it with several open-source subject programs, including programs from OpenSSL and Linux GDK libraries. Our evaluation effectively reveals cache side-channel leakage in such real-world programs. We also empirically show that our test generator is more effective in revealing cache side-channel leakage than traditional fuzz testing tools Radamsa and AFL.","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"88 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134283188","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"EarthCube Software Testing and Assessment Framework","authors":"E. Law","doi":"10.1109/ICSTW.2017.67","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.67","url":null,"abstract":"EarthCube was initiated by the National Science Foundation (NSF) to transform geoscience research by developing cyberinfrastructure to improve access, sharing, visualization, and analysis of all forms of geosciences data and related resources. EarthCube software test-bed project was awarded to create a technology and software validation framework for integration, testing and demonstrations. This paper will provide an overview and characteristics of the framework and detail the guidelines used to support testing and assessment of EarthCube software.","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"7 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128929709","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Shaojiang Wang, Tianyong Wu, Y. Yao, Beihong Jin, Liping Ding
{"title":"Combinatorial Testing on MP3 for Audio Players","authors":"Shaojiang Wang, Tianyong Wu, Y. Yao, Beihong Jin, Liping Ding","doi":"10.1109/ICSTW.2017.48","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.48","url":null,"abstract":"Audio players form a type of frequently used software, in which there are still bugs. In this paper, we investigate the test on MP3 for audio player, a mainstream audio format. An MP3 file consists of ID3v2 tag and MPEG layer III frames, with which there are standards for them. We first introduce the standards through the official documents, then summarize several potentially error-prone points for audio players. For our purpose, we use two black-box test design methods, combinatorial testing namely (CT) and random testing (RT). We test 10 popular audio players and 4 web browser built-in players for supporting audio tag in html5 with these test cases and find 7 types of bugs. Our experimental results show that, although there are standards for ID3v2 tag and MPEG frames, different audio player implementors have different understanding and implementations. We also compare the bugs detected by the test cases with CT and RT.","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"6 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131991829","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"How Good Are Your Types? Using Mutation Analysis to Evaluate the Effectiveness of Type Annotations","authors":"Rahul Gopinath, Eric Walkingshaw","doi":"10.1109/ICSTW.2017.28","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.28","url":null,"abstract":"Software engineers primarily use two orthogonal means to reduce susceptibility to faults: software testing and static type checking. While many strategies exist to evaluate the effectiveness of a test suite in catching bugs, there are few that evaluate the effectiveness of type annotations in a program. This problem is most relevant in the context of gradual or optional typing, where programmers are free to choose which parts of a program to annotate and in what detail. Mutation analysis is one strategy that has proven useful for measuring test suite effectiveness by emulating potential software faults. We propose that mutation analysis can be used to evaluate the effectiveness of type annotations too. We analyze mutants produced by the MutPy mutation framework against both a test suite and against type-annotated programs. We show that, while mutation analysis can be useful for evaluating the effectiveness of type annotations, we require stronger mutation operators that target type information in programs to be an effective mutation analysis tool.","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"40 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125252523","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}