Software Testing, Verification and Reliability最新文献

{"title":"Test code evolution and mutation testing","authors":"Yves Le Traon, Tao Xie","doi":"10.1002/stvr.1877","DOIUrl":"https://doi.org/10.1002/stvr.1877","url":null,"abstract":"<p>In this issue, we are pleased to present two papers on test code evolution and mutation testing, respectively.</p>\u0000<p>The first paper, “Towards automatically identifying the co-change of production and test code” by Yuan Huang, Zhicao Tang, Xiangping Chen and Xiaocong Zhou, presents a method named Jtup that uses machine learning to identify the cochange of production and test code. When a developer makes modifications to a class in production code, Jtup analyses the modified class and determines whether its corresponding test class needs to be modified as well. For machine learning, Jtup incorporates three types of features (code change features, code complexity features and code semantic features). The experimental results show the superior performance of Jtup in both within-project and multiclassification settings, surpassing multiple competing methods (Recommended by Wing Kwong Chan).</p>\u0000<p>The second paper, “A new perspective on the competent programmer hypothesis through the reproduction of real faults with repeated mutations” by Zaheed Ahmed, Eike Schwass, Steffen Herbold, Fabian Trautsch and Jens Grabowski, presents a study of the competent programmer hypothesis based on the ability to reproduce faults through mutation operators. In contrast, previous work only considered how many tokens are changed by bugs or manually compared mutations with faults. The authors reframe the problem of transforming a correct into a buggy AST as a path search problem, where each step of a path is a mutation. The study results support the competent programmer hypothesis and also show that mutation operators are often not in line with the slight differences in correct code introduced by developers (Recommended by Marcio Delamaro).</p>\u0000<p>We hope that these papers will inspire further research in related directions.</p>","PeriodicalId":501413,"journal":{"name":"Software Testing, Verification and Reliability","volume":"6 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-04-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140575262","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"SafeNet: Towards mitigating replaceable unsafe Rust code via a recommendation‐based approach","authors":"Yan Dong, Zhicong Zhang, Mohan Cui, Hui Xu","doi":"10.1002/stvr.1875","DOIUrl":"https://doi.org/10.1002/stvr.1875","url":null,"abstract":"Rust is a system‐level programming language with advantages in memory safety. It ensures that any Rust programs without unsafe code should not incur undefined behaviours. However, unsafe code still plays an essential role in Rust to achieve low‐level control. Therefore, a major design pattern of Rust programs is interior unsafe, which wraps unsafe code as safe APIs and handles all undefined behaviours internally. Rust standard library already provides a rich set of safe APIs to facilitate Rust code development. Nevertheless, due to unfamiliarity with these APIs, developers may misuse unnecessary unsafe code and suffer memory‐safety risks. In this paper, we investigate an approach to mitigate replaceable unsafe code. We first analyse unsafe APIs of the Rust standard library and summarize their common usage patterns. Each pattern corresponds to one or several code samples in our knowledge base. Then, we develop an approach to automatically recognize the usage pattern and recommend corresponding code samples. Our approach leverages dataflow analysis to exclude impossible patterns and employs a BERT‐based machine learning model to find the most similar pattern among the rest. We have conducted evaluation experiments with 472 unsafe code snippets collected from GitHub projects and successfully recognized the pattern of 394 snippets. We hope our approach can assist developers in detecting unnecessary unsafe code and suggesting safe alternatives.","PeriodicalId":501413,"journal":{"name":"Software Testing, Verification and Reliability","volume":"18 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-03-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140018140","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A new perspective on the competent programmer hypothesis through the reproduction of real faults with repeated mutations","authors":"Zaheed Ahmed, Eike Schwass, Steffen Herbold, Fabian Trautsch, Jens Grabowski","doi":"10.1002/stvr.1874","DOIUrl":"https://doi.org/10.1002/stvr.1874","url":null,"abstract":"The competent programmer hypothesis is one of the fundamental assumptions of mutation testing, which claims that most programmers are competent enough to create correct or almost correct source code. This implies that faults should usually manifest through small variations of the correct code. Consequently, researchers assumed that the synthetic faults injected in source code through the mutation operators closely resemble the real faults. Unfortunately, it is still unclear whether the competent programmer hypothesis holds, as past research presents contradictory claims. Within this article, we provide a new perspective on the competent programmer hypothesis and its relation to mutation testing. We try to re-create real-world faults through chains of mutations to understand if there is a direct link between mutation testing and faults. The lengths of these paths help us to understand if the source code is really almost correct, or if large variations are required. Our experiments used a state-of-the-art benchmark database of real faults named Defects4J 2.0.0. It contains 835 reproducible real-world faults in 17 open-source projects that comprise a total of 1044 bug-fix pairs of files. Our results indicate that while the competent programmer hypothesis seems to be true, mutation testing is missing important operators to generate representative real-world faults.","PeriodicalId":501413,"journal":{"name":"Software Testing, Verification and Reliability","volume":"70 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-02-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140025770","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Bud hunting with directed fuzz testing and source code vulnerability detection with advanced graph neural networks","authors":"Yves Le Traon, Tao Xie","doi":"10.1002/stvr.1876","DOIUrl":"https://doi.org/10.1002/stvr.1876","url":null,"abstract":"<p>In this edition, we present two papers that offer significant contributions related to fuzz testing on one hand and vulnerability detection on the other hand, respectively, delving into directed greybox fuzzing (DGF) and tensor-based gated graph neural networks for automatic vulnerability detection in source code.</p>\u0000<p>The first paper, ‘Greybox fuzzing, a scalable and practical approach for software testing’, by Pengfei Wang, Xu Zhou, Tai Yue, Peihong Lin, Yingying Liu and Kai Lu, proposes to go improve greybox fuzzing tools to uncover bugs, with directed greybox fuzzing (DGF). DFG emerges as a strategic alternative to undirected coverage-guided approaches, by allocating its resources purposefully, targeting specific zones like bug-prone areas. This makes DGF particularly effective for patch testing, bug reproduction and specialized bug detection scenarios. The paper conducts a comprehensive study, analysing 42 state-of-the-art fuzzers closely related to DGF. By categorizing DGF into location-directed and behaviour-directed types, the authors unveil its benefits, limitations and potential research avenues. This work not only provides a snapshot of the current state of DGF but also identifies gaps and proposes areas for future investigation.</p>\u0000<p>The second paper, entitled ‘Tensor-based gated graph neural network for automatic vulnerability detection in source code’, is embracing the issue of the rapid expansion of smart devices that intensifies the demand for robust vulnerability detection in source code. Jia Yang, Ou Ruan and JiXin Zhang address this overall challenge by proposing a tensor-based gated graph neural network, named TensorGNN, for function-level vulnerability detection in source code. TensorGNN treats codes as graphs with node features by combining different code graph representations, leading to an accurate code embeddings. The TensorGNN model outperforms existing state-of-the-art works in terms of accuracy and F1 for vulnerability detection across various open-source code corpora. Notably, it achieves these results with significantly fewer training parameters and reduced training time. By introducing a novel perspective to vulnerability detection, this paper opens avenues for further exploration in the intersection of tensor technology and software security.</p>\u0000<p>In conclusion, these two different papers contribute to complementary facets of software quality improvement. As STVR navigates the complexities of deploying safe and secure software, I wish you a pleasant reading that may inspire follow-up research in these two directions.</p>","PeriodicalId":501413,"journal":{"name":"Software Testing, Verification and Reliability","volume":"77 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-02-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139920161","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Investigating the impact of transient hardware faults on deep learning neural network inference","authors":"Md Hasanur Rahman, Sabuj Laskar, Guanpeng Li","doi":"10.1002/stvr.1873","DOIUrl":"https://doi.org/10.1002/stvr.1873","url":null,"abstract":"Safety-critical applications, such as autonomous vehicles, healthcare, and space applications, have witnessed widespread deployment of deep neural networks (DNNs). Inherent algorithmic inaccuracies have consistently been a prevalent cause of misclassifications, even in modern DNNs. Simultaneously, with an ongoing effort to minimize the footprint of contemporary chip design, there is a continual rise in the likelihood of transient hardware faults in deployed DNN models. Consequently, researchers have wondered the extent to which these faults contribute to DNN misclassifications compared to algorithmic inaccuracies. This article delves into the impact of DNN misclassifications caused by transient hardware faults and intrinsic algorithmic inaccuracies in safety-critical applications. Initially, we enhance a cutting-edge fault injector, <span>TensorFI</span>, for TensorFlow applications to facilitate fault injections on modern DNN non-sequential models in a scalable manner. Subsequently, we analyse the DNN-inferred outcomes based on our defined safety-critical metrics. Finally, we conduct extensive fault injection experiments and a comprehensive analysis to achieve the following objectives: (1) investigate the impact of different target class groupings on DNN failures and (2) pinpoint the most vulnerable bit locations within tensors, as well as DNN layers accountable for the majority of safety-critical misclassifications. Our findings regarding different grouping formations reveal that failures induced by transient hardware faults can have a substantially greater impact (with a probability up to 4\u0000<math altimg=\"urn:x-wiley:stvr:media:stvr1873:stvr1873-math-0001\" display=\"inline\" location=\"graphic/stvr1873-math-0001.png\">\u0000<semantics>\u0000<mrow>\u0000<mo>×</mo>\u0000</mrow>\u0000$$ times $$</annotation>\u0000</semantics></math> higher) on safety-critical applications compared to those resulting from algorithmic inaccuracies. Additionally, our investigation demonstrates that higher order bit positions in tensors, as well as initial and final layers of DNNs, necessitate prioritized protection compared to other regions.","PeriodicalId":501413,"journal":{"name":"Software Testing, Verification and Reliability","volume":"11 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-02-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139680151","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Delta4Ms: Improving mutation-based fault localization by eliminating mutant bias","authors":"Hengyuan Liu, Zheng Li, Baolong Han, Yangtao Liu, Xiang Chen, Yong Liu","doi":"10.1002/stvr.1872","DOIUrl":"https://doi.org/10.1002/stvr.1872","url":null,"abstract":"Fault localization is a complex, costly and time-consuming task in software debugging. Numerous automated techniques have been developed to expedite this process. Mutation-based fault localization (MBFL) is one of the most widely studied techniques which uses mutation analysis to generate mutants for revealing potential faults in the program. However, our theoretical analysis exposes an inherent conflict between the fundamental assumption and the essential meaning of existing MBFL suspiciousness. This conflict is caused by mutant bias. Intuitively, the suspiciousness can be corrected by eliminating the mutant bias for more accurately measuring the faulty probability of the corresponding mutant statement. In this paper, we introduce Delta4Ms, a fault localization approach designed to eliminate mutant bias. Delta4Ms integrates the principles of signal theory, modelling the actual suspiciousness and mutant bias as the desired and false signal components, respectively. Based on theoretical derivation, the average suspiciousness of mutants serves as an estimate of mutant bias. Delta4Ms effectively mitigates mutant bias, extracting the desired signal and yielding corrected suspiciousness for fault localization. To precisely estimate mutant bias, higher order mutants (HOMs) are incorporated. We conduct an extensive experimental evaluation of Delta4Ms on 320 real-fault programs from Codeflaws. The results indicate that our model significantly outperforms existing SBFL and MBFL techniques, showing a considerable improvement in fault localization effectiveness. We further assessed the robustness of Delta4Ms by examining different HOM ratios and HOM generation strategies. Moreover, Delta4Ms achieves a substantial reduction in mutation execution cost and minimal accuracy loss through the implementation of test case reduction. Finally, we perform preliminary experiments on 15 real-fault programs from the Defects4J benchmark to assess the generalization of the model's fault localization effectiveness.","PeriodicalId":501413,"journal":{"name":"Software Testing, Verification and Reliability","volume":"5 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-01-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139474771","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Towards automatically identifying the co-change of production and test code","authors":"Yuan Huang, Zhicao Tang, Xiangping Chen, Xiaocong Zhou","doi":"10.1002/stvr.1870","DOIUrl":"https://doi.org/10.1002/stvr.1870","url":null,"abstract":"In software evolution, keeping the test code co-change with the production code is important, because the outdated test code may not work and is ineffective in revealing faults in the production code. However, due to the tight development time, the production and test code may not be co-changed immediately by developers. For example, we analysed the top 1003 popular Java projects on GitHub and found that nearly 9.3% of cases (i.e., 464,417) did not update their production and test code at the same time, that is, the production code is updated first, and then the test code is updated at intervals. The result indicates that much test code will not be updated in time. In this paper, we propose a novel approach, Jtup, to remind developers to co-change the production code and test code in time. Specifically, we first define the co-changed production and test code as a positive instance, while unchanged test code (i.e., production code changed and test code unchanged) as a negative instance. Then, we extract multidimensional features from the production code to characterize the possibility of their co-change, including code change features, code complexity features, and code semantic features. Finally, several machine learning-based methods are employed to identify the co-changed production and test code. We conduct comprehensive experiments on 20 datasets, and the results show that the Accuracy, Precision, and Recall achieved by Jtup are 76.7%, 78.1%, and 77.4%, which outperforms the state-of-the-art method.","PeriodicalId":501413,"journal":{"name":"Software Testing, Verification and Reliability","volume":"60 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-01-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139465249","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Test case prioritization and mutation testing","authors":"Yves Le Traon, Tao Xie","doi":"10.1002/stvr.1871","DOIUrl":"https://doi.org/10.1002/stvr.1871","url":null,"abstract":"<p>In this issue, we are pleased to present two papers on test case prioritization and mutation testing, respectively.</p>\u0000<p>The first paper, ‘Semantic-aware two-phase test case prioritization for continuous integration’ by Yingling Li, Ziao Wang, Junjie Wang, Jie Chen, Rui Mou, and Guibing Li, presents the SatTCP framework to conduct precise prioritization with low time overhead, in order to improve the cost effectiveness of typical continuous integration (CI) testing with frequent code submissions. In SatTCP, coarse-grained filtering based on information retrieval (IR) techniques roughly sorts test cases and selects a certain number of tests for the subsequent prioritization; then fine-grained prioritization based on pretrained Siamese network conducts precise prioritization of initially ranked test sets. The evaluation results show that SatTCP outperforms all the baselines under comparison, and achieves the lowest test costs. (Recommended by Yves Le Traon).</p>\u0000<p>The second paper, ‘Mutation testing optimisations using the Clang front-end’ by Sten Vercammen, Serge Demeyer, Markus Borg, Niklas Pettersson, and Görel Hedin, presents an investigation to which extent the Clang front-end and its state-of-the-art program analysis facilities allow to implement existing strategies for mutation optimization within the C language family. The authors develop a proof-of-concept tool used to collect detailed measurements for each mutation phase. The authors conduct evaluation of the proof-of-concept tool on four open-source C++ libraries and one industrial component. The evaluation results show that the ‘Generate Mutants’ and ‘Detect (Un)Reachable Mutants’ steps are for all practical purposes negligible; the ‘Compile Mutants’ step takes a significant amount of time and the compilation of the invalid and unreachable mutants is considerable; the ‘Execute Mutants’ step is the other dominant factor. (Recommended by Mike Papadakis).</p>\u0000<p>We hope that these papers will inspire further research in related directions.</p>","PeriodicalId":501413,"journal":{"name":"Software Testing, Verification and Reliability","volume":"171 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-12-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138681353","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"The progress, challenges, and perspectives of directed greybox fuzzing","authors":"Pengfei Wang, Xu Zhou, Tai Yue, Peihong Lin, Yingying Liu, Kai Lu","doi":"10.1002/stvr.1869","DOIUrl":"https://doi.org/10.1002/stvr.1869","url":null,"abstract":"Greybox fuzzing is a scalable and practical approach for software testing. Most greybox fuzzing tools are coverage-guided as reaching high code coverage is more likely to find bugs. However, since most covered codes may not contain bugs, blindly extending code coverage is less efficient, especially for corner cases. Unlike coverage-guided greybox fuzzing which increases code coverage in an undirected manner, directed greybox fuzzing (DGF) spends most of its time allocation on reaching specific targets (e.g. the bug-prone zone) without wasting resources stressing unrelated parts. Thus, DGF is particularly suitable for scenarios such as patch testing, bug reproduction, and special bug detection. For now, DGF has become an active research area. However, DGF has general limitations and challenges that are worth further studying. Based on the investigation of 42 state-of-the-art fuzzers that are closely related to DGF, we conducted the first in-depth study to summarize the empirical evidence on the research progress of DGF. This paper studies DGF from a broader view, which takes into account not only the location-directed type that targets specific code parts but also the behavior-directed type that aims to expose abnormal program behaviors. By analyzing the benefits and limitations of DGF research, we try to identify gaps in current research, meanwhile, reveal new research opportunities and suggest areas for further investigation.","PeriodicalId":501413,"journal":{"name":"Software Testing, Verification and Reliability","volume":"105 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-12-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138715825","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Tensor-based gated graph neural network for automatic vulnerability detection in source code","authors":"Jia Yang, Ou Ruan, JiXin Zhang","doi":"10.1002/stvr.1867","DOIUrl":"https://doi.org/10.1002/stvr.1867","url":null,"abstract":"The rapid expansion of smart devices leads to the increasing demand for vulnerability detection in the cyber security field. Writing secure source codes is crucial to protect applications and software. Recent vulnerability detection methods are mainly using machine learning and deep learning. However, there are still some challenges, how to learn accurate source code semantic embedding at the function level, how to effectively perform vulnerability detection using the learned semantic embedding of source code and how to solve the overfitting problem of learning-based models. In this paper, we consider codes as various graphs with node features and propose a tensor-based gated graph neural network called TensorGNN to produce code embedding for function-level vulnerability detection. First, we propose a high-dimensional tensor for combining different code graph representations. Second, inspired by the work of tensor technology, we propose the TensorGNN model to produce accurate code representations using the graph tensor. We evaluate our model on 7 C and C++ large open-source code corpus (e.g. SARD&amp;NVD, Debian, SATE IV, FFmpeg, libpng&amp;LibTiff, Wireshark and Github datasets), which contains 13 types of vulnerabilities. Our TensorGNN model improves on existing state-of-the-art works by 10%–30% on average in terms of vulnerability detection accuracy and F1, while our TensorGNN model needs less training time and model parameters. Specifically, compared with other existing works, our model reduces 25–47 times of the number of parameters and decreases 3–10 times of training time. Results of evaluations show that TensorGNN has better performance while using fewer training parameters and less training time.","PeriodicalId":501413,"journal":{"name":"Software Testing, Verification and Reliability","volume":"6 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-11-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138528120","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}