Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis最新文献_第5页

Mitigating the effects of flaky tests on mutation testing 减轻薄片测试对突变测试的影响

Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis Pub Date : 2019-07-10 DOI: 10.1145/3293882.3330568

A. Shi, Jonathan Bell, D. Marinov

{"title":"Mitigating the effects of flaky tests on mutation testing","authors":"A. Shi, Jonathan Bell, D. Marinov","doi":"10.1145/3293882.3330568","DOIUrl":"https://doi.org/10.1145/3293882.3330568","url":null,"abstract":"Mutation testing is widely used in research as a metric for evaluating the quality of test suites. Mutation testing runs the test suite on generated mutants (variants of the code under test), where a test suite kills a mutant if any of the tests fail when run on the mutant. Mutation testing implicitly assumes that tests exhibit deterministic behavior, in terms of their coverage and the outcome of a test (not) killing a certain mutant. Such an assumption does not hold in the presence of flaky tests, whose outcomes can non-deterministically differ even when run on the same code under test. Without reliable test outcomes, mutation testing can result in unreliable results, e.g., in our experiments, mutation scores vary by four percentage points on average between repeated executions, and 9% of mutant-test pairs have an unknown status. Many modern software projects suffer from flaky tests. We propose techniques that manage flakiness throughout the mutation testing process, largely based on strategically re-running tests. We implement our techniques by modifying the open-source mutation testing tool, PIT. Our evaluation on 30 projects shows that our techniques reduce the number of \"unknown\" (flaky) mutants by 79.4%.","PeriodicalId":20624,"journal":{"name":"Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis","volume":"208 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2019-07-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"88183955","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 36

A large-scale study of application incompatibilities in Android Android应用程序不兼容性的大规模研究

Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis Pub Date : 2019-07-10 DOI: 10.1145/3293882.3330564

Haipeng Cai, Ziyi Zhang, Li Li, Xiaoqin Fu

{"title":"A large-scale study of application incompatibilities in Android","authors":"Haipeng Cai, Ziyi Zhang, Li Li, Xiaoqin Fu","doi":"10.1145/3293882.3330564","DOIUrl":"https://doi.org/10.1145/3293882.3330564","url":null,"abstract":"The rapid expansion of the Android ecosystem is accompanied by continuing diversification of platforms and devices, resulting in increasing incompatibility issues which damage user experiences and impede app development productivity. In this paper, we conducted a large-scale, longitudinal study of compatibility issues in 62,894 benign apps developed in the past eight years, to understand the symptoms and causes of these issues. We further investigated the incompatibilities that are actually exercised at runtime through the system logs and execution traces of 15,045 apps. Our study revealed that, among others, (1) compatibility issues were prevalent and persistent at both installation and run time, with greater prevalence of run-time incompatibilities, (2) there were no certain Android versions that consistently saw more or less app incompatibilities than others, (3) installation-time incompatibilities were strongly correlated with the minSdkVersion specified in apps, while run-time incompatibilities were most significantly correlated with the underlying platform’s API level, and (4) installation-time incompatibilities were mostly due to apps’ use of architecture-incompatible native libraries, while run-time incompatibilities were mostly due to API changes during SDK evolution. We offered further insights into app incompatibilities, as well as recommendations on dealing with the issues for bother developers and end users of Android apps.","PeriodicalId":20624,"journal":{"name":"Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis","volume":"388 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2019-07-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"84989592","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 36

Assessing the state and improving the art of parallel testing for C 评估C语言并行测试的现状并改进其技术

Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis Pub Date : 2019-07-10 DOI: 10.1145/3293882.3330573

Oliver Schwahn, Nicolas Coppik, Stefan Winter, N. Suri

{"title":"Assessing the state and improving the art of parallel testing for C","authors":"Oliver Schwahn, Nicolas Coppik, Stefan Winter, N. Suri","doi":"10.1145/3293882.3330573","DOIUrl":"https://doi.org/10.1145/3293882.3330573","url":null,"abstract":"The execution latency of a test suite strongly depends on the degree of concurrency with which test cases are executed. However, if test cases are not designed for concurrent execution, they may interfere, causing result deviations compared to sequential execution. To prevent this, each test case can be provided with an isolated execution environment, but the resulting overheads diminish the merit of parallel testing. Our large-scale analysis of the Debian Buster package repository shows that existing test suites in C projects make limited use of parallelization. We present an approach to (a) analyze the potential of C test suites for safe concurrent execution, i.e., result invariance compared to sequential execution, and (b) execute tests concurrently with different parallelization strategies using processes or threads if it is found to be safe. Applying our approach to 9 C projects, we find that most of them cannot safely execute tests in parallel due to unsafe test code or unsafe usage of shared variables or files within the program code. Parallel test execution shows a significant acceleration over sequential execution for most projects. We find that multi-threading rarely outperforms multi-processing. Finally, we observe that the lack of a common test framework for C leaves make as the standard driver for running tests, which introduces unnecessary performance overheads for test execution.","PeriodicalId":20624,"journal":{"name":"Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis","volume":"88 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2019-07-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"89368244","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 11

Deferred concretization in symbolic execution via fuzzing 通过模糊处理实现符号执行中的延迟具体化

Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis Pub Date : 2019-07-10 DOI: 10.1145/3293882.3330554

Awanish Pandey, Phani Raj Goutham Kotcharlakota, Subhajit Roy

引用次数: 17

Theory and practice of string solvers (invited talk abstract) 弦解法的理论与实践(特邀演讲摘要)

Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis Pub Date : 2019-07-10 DOI: 10.1145/3293882.3338993

Adam Kiezun, Philip J. Guo, Pieter Hooimeijer, Michael D. Ernst, Vijay Ganesh

{"title":"Theory and practice of string solvers (invited talk abstract)","authors":"Adam Kiezun, Philip J. Guo, Pieter Hooimeijer, Michael D. Ernst, Vijay Ganesh","doi":"10.1145/3293882.3338993","DOIUrl":"https://doi.org/10.1145/3293882.3338993","url":null,"abstract":"The paper titled \"Hampi: A Solver for String Constraints\" was published in the proceedings of the International Symposium on Software Testing and Analysis (ISSTA) 2009, and has been selected to receive the ISSTA 2019 Impact Paper Award. The paper describes HAMPI, one of the first practical solver aimed at solving the satisfiability problem for a theory of string (word) equations, operations over strings, predicates over regular expressions and context-free grammars. HAMPI has been used widely to solve many software engineering and security problems, and has inspired considerable research on string solving algorithms and their applications. In this talk, we review the state of research on the theory and practice of string solving algorithms, specifically highlighting key historical developments that have led to their widespread use. On the practical front, we discuss different kinds of algorithmic paradigms, such as word- and automata-based, that have been developed to solve string and regular expression constraints. We then focus on the many hardness results that theorists have proved for fragments of theories over strings. Finally, we conclude with open theoretical problems, practical algorithmic challenges, and future applications of string solvers.","PeriodicalId":20624,"journal":{"name":"Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis","volume":"2 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2019-07-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"72637422","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

Continuous software performance assessment: detecting performance problems of software libraries on every build 持续的软件性能评估:在每次构建中检测软件库的性能问题

Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis Pub Date : 2019-07-10 DOI: 10.1145/3293882.3338982

Christoph Laaber

引用次数: 7

Failure clustering without coverage 没有覆盖的故障聚类

Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis Pub Date : 2019-07-10 DOI: 10.1145/3293882.3330561

Mojdeh Golagha, Constantin Lehnhoff, A. Pretschner, H. Ilmberger

{"title":"Failure clustering without coverage","authors":"Mojdeh Golagha, Constantin Lehnhoff, A. Pretschner, H. Ilmberger","doi":"10.1145/3293882.3330561","DOIUrl":"https://doi.org/10.1145/3293882.3330561","url":null,"abstract":"Developing and integrating software in the automotive industry is a complex task and requires extensive testing. An important cost factor in testing and debugging is the time required to analyze failing tests. In the context of regression testing, usually, large numbers of tests fail due to a few underlying faults. Clustering failing tests with respect to their underlying faults can, therefore, help in reducing the required analysis time. In this paper, we propose a clustering technique to group failing hardware-in-the-loop tests based on non-code-based features, retrieved from three different sources. To effectively reduce the analysis effort, the clustering tool selects a representative test for each cluster. Instead of analyzing all failing tests, testers only inspect the representative tests to find the underlying faults. We evaluated the effectiveness and efficiency of our solution in a major automotive company using 86 regression test runs, 8743 failing tests, and 1531 faults. The results show that utilizing our clustering tool, testers can reduce the analysis time more than 60% and find more than 80% of the faults only by inspecting the representative tests.","PeriodicalId":20624,"journal":{"name":"Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis","volume":"30 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2019-07-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"85195697","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 12

Automated API-usage update for Android apps Android应用程序的自动api使用更新

Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis Pub Date : 2019-07-10 DOI: 10.1145/3293882.3330571

M. Fazzini, Qi Xin, A. Orso

{"title":"Automated API-usage update for Android apps","authors":"M. Fazzini, Qi Xin, A. Orso","doi":"10.1145/3293882.3330571","DOIUrl":"https://doi.org/10.1145/3293882.3330571","url":null,"abstract":"Mobile apps rely heavily on the application programming interface (API) provided by their underlying operating system (OS). Because OS and API can change frequently, developers must quickly update their apps to ensure that the apps behave as intended with new API and OS versions. To help developers with this tedious, error prone, and time consuming task, we developed a technique that can automatically perform app updates for API changes based on examples of how other developers evolved their apps for the same changes. Given a target app to be updated and information about the changes in the API, our technique performs four main steps. First, it analyzes the target app to identify code affected by API changes. Second, it searches existing code bases for examples of updates to the new version of the API. Third, it analyzes, ranks, and transforms into generic patches the update examples found in the previous step. Finally, it applies the generated patches to the target app in order of ranking, while performing differential testing to validate the update. We implemented our technique and performed an empirical evaluation on 15 real-world apps with promising results. Overall, our technique was able to update 85% of the API changes considered and automatically validate 68% of the updates performed.","PeriodicalId":20624,"journal":{"name":"Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis","volume":"93 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2019-07-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"90432062","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 51

Mining constraints for grammar fuzzing 挖掘语法模糊的约束

Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis Pub Date : 2019-07-10 DOI: 10.1145/3293882.3338983

Michaël Mera

引用次数: 7

A cost-effective strategy for software vulnerability prediction based on bellwether analysis 基于领头羊分析的高效软件漏洞预测策略

Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis Pub Date : 2019-07-10 DOI: 10.1145/3293882.3338985

P. Kudjo, Jinfu Chen

{"title":"A cost-effective strategy for software vulnerability prediction based on bellwether analysis","authors":"P. Kudjo, Jinfu Chen","doi":"10.1145/3293882.3338985","DOIUrl":"https://doi.org/10.1145/3293882.3338985","url":null,"abstract":"Vulnerability Prediction Models (VPMs) aims to identify vulnerable and non-vulnerable components in large software systems. Consequently, VPMs presents three major drawbacks (i) finding an effective method to identify a representative set of features from which to construct an effective model. (ii) the way the features are utilized in the machine learning setup (iii) making an implicit assumption that parameter optimization would not change the outcome of VPMs. To address these limitations, we investigate the significant effect of the Bellwether analysis on VPMs. Specifically, we first develop a Bellwether algorithm to identify and select an exemplary subset of data to be considered as the Bellwether to yield improved prediction accuracy against the growing portfolio benchmark. Next, we build a machine learning approach with different parameter settings to show the improvement of performance of VPMs. The prediction results of the suggested models were assessed in terms of precision, recall, F-measure, and other statistical measures. The preliminary result shows the Bellwether approach outperforms the benchmark technique across the applications studied with F-measure values ranging from 51.1%-98.5%.","PeriodicalId":20624,"journal":{"name":"Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis","volume":"63 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2019-07-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"84504467","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 9