J. Ayerdi, Aitor Arrieta, E. Pobee, Maite Arratibel
{"title":"Multi-Objective Metamorphic Test Case Selection: an Industrial Case Study (Practical Experience Report)","authors":"J. Ayerdi, Aitor Arrieta, E. Pobee, Maite Arratibel","doi":"10.1109/ISSRE55969.2022.00058","DOIUrl":"https://doi.org/10.1109/ISSRE55969.2022.00058","url":null,"abstract":"Metamorphic testing is a technique that has shown great potential to alleviate the test oracle problem by exploiting the relations among the inputs and outputs of different executions of a system. However, this approach requires multiple test executions. In applications like Cyber-Physical Systems (CPSs), where the test executions can be very expensive in terms of time and resources needed, this can supose a problem. Therefore, it is paramount to optimize the test suite to reduce the costs of verifying the system. Test case selection is an optimization technique which accomplishes this by selecting a subset of test cases while aiming to preserve the effectiveness of the original test suite as much as possible. While there are many approaches for test case selection in the existing literature, none of them has been proposed for the metamorphic test case selection problem, where each metamorphic test case consists of a source and, at least, a follow-up test case pair. In this work, we present an evolutionary multi-objective approach for the metamorphic test case selection problem, adapting existing multi-objective test selection techniques and proposing new evolutionary operators and objective functions. Further-more, we evaluate our approach with a set of metamorphic tests developed for an industrial case study from the elevation domain. The results suggest that our approach outperforms both Random Search and the same metaheuristic algorithm without the new evolutionary operators we propose.","PeriodicalId":373322,"journal":{"name":"2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE)","volume":"39 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122560488","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"CEMENT: On the Use of Evolutionary Coupling Between Tests and Code Units. A Case Study on Fault Localization","authors":"Jeongju Sohn, Mike Papadakis","doi":"10.1109/ISSRE55969.2022.00023","DOIUrl":"https://doi.org/10.1109/ISSRE55969.2022.00023","url":null,"abstract":"Many software engineering techniques, such as fault localization, operate based on relevance relationships between tests and code. These relationships are often inferred using dynamic test execution information that approximates the link between relevant code units and asserted, by the tests, program behaviour. Unfortunately, in practice, dynamic information is not always available due to the instrumentation overheads or the nature of the production environments. This paper proposes CEMENT, a static technique that automatically infers such test and code relationships given the projects' evolution. The key idea is that developers make relevant changes to tests and code units around the same period, and thus their co-evolution reflects probable links between them. We evaluate CEMENT on 15 open-source projects, showing that it captures relevant links. We further conduct a fault localization case study and demonstrate that CEMENT can perform comparably to a current Information Retrieval-based Fault Localization (IRFL) approach. Additional analysis reveals that CEMENT and the IRFL localize different faults, suggesting complementarity. We leverage this complementarity by combining these two through voting, identifying 82% (32) of the faults located by at least one of them. These results suggest that test and code evolutionary coupling can effectively support test and debugging activities.","PeriodicalId":373322,"journal":{"name":"2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129765432","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Bootstrapping confidence in future safety from past safe operation","authors":"P. Bishop, A. Povyakalo, L. Strigini","doi":"10.1109/ISSRE55969.2022.00020","DOIUrl":"https://doi.org/10.1109/ISSRE55969.2022.00020","url":null,"abstract":"We propose an approach to managing the rolling out of a new system type so as to contain the risk of mishaps in its operation to an acceptable level, while using the evidence of safe operation to support confidence for extending the scale of operation. This cautious approach of “bootstrapping” confidence in the safety of a system is now widely applied to autonomous vehicle (AVs), our example. AVs are subject to extreme safety requirements; a major concern is the inability to give meaningful quantitative assurance of safety of an AV type, to the extent required by society, before it is used extensively. We exploit a previously published approach to achieving more moderate, but useful, assurance, e.g. about low enough probability of causing accidents in a limited period of operation; and demonstrate how this approach supports choice of production/deployment strategies, so as to manage the growth of a fleet of AVs for a given accepted level of risk. Via a formal mathematical description of “confidence bootstrapping”, we show: (1) that it is a substantially sound approach in the right circumstances, and useful for deciding about the early deployment phase for a new system; (2) how much confidence can be rightly derived from such a “cautious deployment” approach, avoiding over-optimism; (3) under which conditions our sound formulas for future confidence are applicable; (4) thus, which analyses of the concrete situations, and/or constraints on practice, are needed in order to enjoy the advantages of provably correct confidence in adequate future safety over a definite time (“confidence horizon”).","PeriodicalId":373322,"journal":{"name":"2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE)","volume":"38 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124758559","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
B. Du, Yuxiaoyang Cai, Haifeng Wang, Y. Liu, Xiang Chen
{"title":"Improving the Performance of Mutation-based Fault Localization via Mutant Bias Practical Experience Report","authors":"B. Du, Yuxiaoyang Cai, Haifeng Wang, Y. Liu, Xiang Chen","doi":"10.1109/ISSRE55969.2022.00038","DOIUrl":"https://doi.org/10.1109/ISSRE55969.2022.00038","url":null,"abstract":"Mutation-Based Fault Localization (MBFL) is one of the most widely studied techniques. MBFL adopts mutation analysis to generate mutants for revealing potential faults in the program. Previous studies proposed approaches to optimize MBFL in terms of efficiency and accuracy. However, these approaches ignored the difference of mutants on correct entities (such as statements) and faulty entities, which we refer this kind of difference as mutant bias. In this study, we identify and analyze the impact of mutant bias on MBFL. We find that the mutant bias may introduce effects to statement suspiciousness and negatively influence the fault localization accuracy of MBFL. To mitigate the mutant bias, we propose Delta4Ms, a model that captures the mutant bias from the mutants of the same statements. Then the real suspiciousness is obtained by removing the bias from the practical suspiciousness. To evaluate the performance of our proposed method, we conduct experimental studies on 320 real-world programs from Codeflaws. The experimental results show that Delta4Ms improves the fault localization accuracy of MBFL. Besides, Delta4Ms outperforms the state-of-the-art SBFL and three MBFL techniques significantly. Moreover, Delta4Ms ranks 94 and 161 of the target faults within the top-5 suspicious statements in single-fault and multiple-fault programs, respectively.","PeriodicalId":373322,"journal":{"name":"2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE)","volume":"488 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117318642","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Resilient Mechanism Against Byzantine Failure for Distributed Deep Reinforcement Learning","authors":"Mingyue Zhang, Zhi Jin, Jian Hou, Renwei Luo","doi":"10.1109/ISSRE55969.2022.00044","DOIUrl":"https://doi.org/10.1109/ISSRE55969.2022.00044","url":null,"abstract":"Distributed deep reinforcement learning(DDRL) has been used in distributed systems to better improve the adaptability. However, DDRL-based systems are also inevitably under the threat of Byzantine workers. There is an urgent need to enhance the resilience of the DDRL-based system against Byzantine failures. This paper proposes a resilient mechanism for mitigating the influence of Byzantine workers on DDRL-based systems. First, we formalize the DDRL-based system as a multi-armed bandit model for well capturing the collective effect of workers on the whole learning process, and then transforming the resilient mechanism design problem into the sampling policy optimization problem. Second, we propose a self-adaptation process for filtering out the harmful data generated by Byzantine workers and theoretically give a mathematical analysis of the understanding, demonstrating its effectiveness under ideal conditions. Third, based on a typical DDRL-based system (i.e., Asynchronous Advantage Actor-Critic, A3C), we implement a resilient distributed A3C (ReD-A3C). With extensive experiments on the DDRL benchmark tasks, we show that ReD-A3C outperforms available Byzantine tolerant approaches.","PeriodicalId":373322,"journal":{"name":"2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131169495","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Share or Not Share? Towards the Practicability of Deep Models for Unsupervised Anomaly Detection in Modern Online Systems","authors":"Zilong He, Pengfei Chen, Tao Huang","doi":"10.1109/ISSRE55969.2022.00014","DOIUrl":"https://doi.org/10.1109/ISSRE55969.2022.00014","url":null,"abstract":"Anomaly detection is crucial in the management of modern online systems. Due to the complexity of patterns in the monitoring data and the lack of labelled data with anomalies, recent studies mainly adopt deep unsupervised models to address this problem. Notably, even though these models have achieved a great success on experimental datasets, there are still several challenges for them to be successfully applied in a real-world modern online system. Such challenges stem from some significant properties of modern online systems, e.g., large scale, diversity and dynamics. This study investigates how these properties affect the adoption of deep anomaly detectors in modern online systems. Furthermore, we claim that model sharing is an effective way to overcome these challenges. To support this claim, we systematically study the feasibility and necessity of model sharing for unsupervised anomaly detection. In addition, we further propose a novel model, Uni-AD, which works well for model sharing. Based upon Transformer encoder layers and Base layers, Uni-AD can effectively model diverse patterns for different monitored entities and further perform anomaly detection accurately. Besides, it can accept variable-length inputs, which is a required property for a model that needs to be shared. Extensive experiments on two real-world large-scale datasets demonstrate the effectiveness and practicality of Uni-AD.","PeriodicalId":373322,"journal":{"name":"2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE)","volume":"76 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134220413","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Ao Liu, Chenkai Guo, Naipeng Dong, Yinjie Wang, Jing Xu
{"title":"DALT: Deep Activity Launching Test via Intent-Constraint Extraction","authors":"Ao Liu, Chenkai Guo, Naipeng Dong, Yinjie Wang, Jing Xu","doi":"10.1109/ISSRE55969.2022.00053","DOIUrl":"https://doi.org/10.1109/ISSRE55969.2022.00053","url":null,"abstract":"The frequent usage of the activity and intent in Android app development makes activity launching communication the focus of app analysis, which inspires the proposal of Activity Launching Test (ALT). Existing static analysis approaches are limited in test case generation and crash triggering of ALT, due to their path-insensitive nature and insufficient intent attribute exploration. This work proposes DALT, an activity test frame-work for launching-related bug detection, which empowers an inter-procedural, context-, flow- and path-sensitive static analysis to generate proper intents as test cases. By tailoring symbolic execution for intent propagation, DALT is able to explore statements in deep code position and extract conditional constraints in diverse launching-related execution paths. Consequently, invalid test paths are substantially reduced via the guidance of the extracted constraints. DALT also supports significantly more intent attribute types and value types of attributes, by reformatting them to be compatible with the commonly used constraint solvers. Extensive comparison experiments have been conducted from diverse validation dimensions. The results demonstrate that compared to the state-of-the-art approach, DALT is more effective in successfully launching activities and detecting bugs hiding in deep positions of the program paths.","PeriodicalId":373322,"journal":{"name":"2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE)","volume":"375 5","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132781079","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Yiming Gan, P. Whatmough, Jingwen Leng, Bo Yu, Shaoshan Liu, Yuhao Zhu
{"title":"Braum: Analyzing and Protecting Autonomous Machine Software Stack","authors":"Yiming Gan, P. Whatmough, Jingwen Leng, Bo Yu, Shaoshan Liu, Yuhao Zhu","doi":"10.1109/ISSRE55969.2022.00019","DOIUrl":"https://doi.org/10.1109/ISSRE55969.2022.00019","url":null,"abstract":"Autonomous machines, such as Autonomous Vehicles (AV), are vulnerable to a variety of different faults such as radiation-induced soft/transient errors, adversarial attacks, and software bugs, which all jeopardize the reliability of autonomous machines. How vulnerable the AV software stack is to different error sources, however, remains an open question. This paper performs comprehensively fault injections to study how the AV software stack behaves under different error sources. We show that algorithms in an AV software stack inherently possess different forms of masking mechanisms. Based on the characteristic of the inherent fault tolerance mechanisms, we formalize the notion of Fault Tolerance Level (FTL), which quantifies how faults in an algorithm can be masked and/or attenuated without affecting the actuator commands, providing opportunities to relax fault protection. Leveraging the FTL formulation, we propose a dynamic protection system, which, at the high level, spends the limited protection budget (e.g., spatial/temporal redundancy) on the most vulnerable parts of the AV software (i.e., with the lowest FTL). Using Autoware as a case study, we show that our system reduces the error rate of AV software stack by more than 90% with negliaible performance overhead.","PeriodicalId":373322,"journal":{"name":"2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE)","volume":"22 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114608618","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jing Su, Zhen Zhang, Peng Wu, Xuran Li, Jian Zhang
{"title":"Adversarial Input Detection Based on Critical Transformation Robustness","authors":"Jing Su, Zhen Zhang, Peng Wu, Xuran Li, Jian Zhang","doi":"10.1109/ISSRE55969.2022.00045","DOIUrl":"https://doi.org/10.1109/ISSRE55969.2022.00045","url":null,"abstract":"Recent studies have shown that ad-hoc image transformations are effective for defending certain adversarial attacks. It is desirable to determine which transformations are more effective than others for adversarial defenses before these transformations are being deployed in practice. We propose in this paper the notion of Critical Transformation Robustness (CTR), which can indicate potentially the detection performance of an input transformation through the difference between its CTR on clean inputs and that on adversarial ones. Then, based on this new notion, we further present a general training framework that can deliver an effective and efficient adversarial detector, which features in a customized combination of specific input transformations for defending the given, possibly mixed, adversarial attacks. We evaluate our training framework on 3 typical image datasets with 17 types of popular input transformations for detecting a mixture of 22 types of adversarial attacks. Experimental results show that the CTR differences between the clean and adversarial inputs can essentially guide the selection of an effective combination of input transformations with their nearly-optimal parameter values. Furthermore, compared with the state-of-the-art input transformation-based adversarial detection methods, the detectors generated by our training framework exhibit on average 73.4% - 87.5% higher performance on the mixed adversarial attacks.","PeriodicalId":373322,"journal":{"name":"2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE)","volume":"2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126144628","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Yiqun Chen, Oliver Schwahn, R. Natella, M. Bradbury, N. Suri
{"title":"SlowCoach: Mutating Code to Simulate Performance Bugs","authors":"Yiqun Chen, Oliver Schwahn, R. Natella, M. Bradbury, N. Suri","doi":"10.1109/ISSRE55969.2022.00035","DOIUrl":"https://doi.org/10.1109/ISSRE55969.2022.00035","url":null,"abstract":"Performance bugs are unnecessarily inefficient code chunks in software codebases that cause prolonged execution times and degraded computational resource utilization. For performance bug diagnostics, tools that aid in the identification of said bugs, such as benchmarks and profilers, are commonly employed. However, due to factors such as insufficient workloads or ineffective benchmarks, software defects related to code inefficiencies are inherently difficult to diagnose. Hence, the capabilities of performance bug diagnostic tools are limited and performance bug instances may be missed. Traditional mutation testing (MT) is a technique for quantifying a test suite's ability to find functional bugs by mutating the code of the test subject. Similarly, we adopt performance mutation testing (PMT) to evaluate performance bug diagnostic tools and identify where improvements need to be made to a performance testing methodology. We carefully investigate the different performance bug fault models and how synthesized performance bugs based on these models can evaluate benchmarks and workload selection to help improve performance diagnostics. In this paper, we present the design of our PMT framework, SLOWCOACH, and evaluate it with over 1600 mutants from 4 real-world software projects.","PeriodicalId":373322,"journal":{"name":"2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE)","volume":"47 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126137222","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}