如果我破坏了代码,测试会告诉我吗?

Rainer Niedermayr, Elmar Jürgens, S. Wagner
{"title":"如果我破坏了代码,测试会告诉我吗?","authors":"Rainer Niedermayr, Elmar Jürgens, S. Wagner","doi":"10.1145/2896941.2896944","DOIUrl":null,"url":null,"abstract":"Automated tests play an important role in software evolution because they can rapidly detect faults introduced during changes. In practice, code-coverage metrics are often used as criteria to evaluate the effectiveness of test suites with focus on regression faults. However, code coverage only expresses which portion of a system has been executed by tests, but not how effective the tests actually are in detecting regression faults. Our goal was to evaluate the validity of code coverage as a measure for test effectiveness. To do so, we conducted an empirical study in which we applied an extreme mutation testing approach to analyze the tests of open-source projects written in Java. We assessed the ratio of pseudo-tested methods (those tested in a way such that faults would not be detected) to all covered methods and judged their impact on the software project. The results show that the ratio of pseudo-tested methods is acceptable for unit tests but not for system tests (that execute large portions of the whole system). Therefore, we conclude that the coverage metric is only a valid effectiveness indicator for unit tests.","PeriodicalId":438234,"journal":{"name":"2016 IEEE/ACM International Workshop on Continuous Software Evolution and Delivery (CSED)","volume":"24 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2016-05-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"24","resultStr":"{\"title\":\"Will My Tests Tell Me If I Break This Code?\",\"authors\":\"Rainer Niedermayr, Elmar Jürgens, S. Wagner\",\"doi\":\"10.1145/2896941.2896944\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Automated tests play an important role in software evolution because they can rapidly detect faults introduced during changes. In practice, code-coverage metrics are often used as criteria to evaluate the effectiveness of test suites with focus on regression faults. However, code coverage only expresses which portion of a system has been executed by tests, but not how effective the tests actually are in detecting regression faults. Our goal was to evaluate the validity of code coverage as a measure for test effectiveness. To do so, we conducted an empirical study in which we applied an extreme mutation testing approach to analyze the tests of open-source projects written in Java. We assessed the ratio of pseudo-tested methods (those tested in a way such that faults would not be detected) to all covered methods and judged their impact on the software project. The results show that the ratio of pseudo-tested methods is acceptable for unit tests but not for system tests (that execute large portions of the whole system). Therefore, we conclude that the coverage metric is only a valid effectiveness indicator for unit tests.\",\"PeriodicalId\":438234,\"journal\":{\"name\":\"2016 IEEE/ACM International Workshop on Continuous Software Evolution and Delivery (CSED)\",\"volume\":\"24 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2016-05-14\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"24\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"2016 IEEE/ACM International Workshop on Continuous Software Evolution and Delivery (CSED)\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1145/2896941.2896944\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"2016 IEEE/ACM International Workshop on Continuous Software Evolution and Delivery (CSED)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1145/2896941.2896944","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 24

摘要

自动化测试在软件发展中扮演着重要的角色,因为它们可以快速检测到变更过程中引入的错误。在实践中,代码覆盖度量标准经常被用作评估测试套件有效性的标准,重点关注回归错误。然而,代码覆盖率只表示测试执行了系统的哪一部分,而不是测试在检测回归错误方面的实际有效性。我们的目标是将代码覆盖率的有效性作为测试有效性的度量来评估。为此,我们进行了一项实证研究,其中我们应用了一种极端突变测试方法来分析用Java编写的开源项目的测试。我们评估了伪测试方法(以一种不会检测到错误的方式测试的方法)与所有覆盖方法的比例,并判断它们对软件项目的影响。结果表明,伪测试方法的比例对于单元测试是可以接受的,但对于系统测试(执行整个系统的大部分)则不行。因此,我们得出结论,覆盖率度量只是单元测试的一个有效的有效性指标。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
Will My Tests Tell Me If I Break This Code?
Automated tests play an important role in software evolution because they can rapidly detect faults introduced during changes. In practice, code-coverage metrics are often used as criteria to evaluate the effectiveness of test suites with focus on regression faults. However, code coverage only expresses which portion of a system has been executed by tests, but not how effective the tests actually are in detecting regression faults. Our goal was to evaluate the validity of code coverage as a measure for test effectiveness. To do so, we conducted an empirical study in which we applied an extreme mutation testing approach to analyze the tests of open-source projects written in Java. We assessed the ratio of pseudo-tested methods (those tested in a way such that faults would not be detected) to all covered methods and judged their impact on the software project. The results show that the ratio of pseudo-tested methods is acceptable for unit tests but not for system tests (that execute large portions of the whole system). Therefore, we conclude that the coverage metric is only a valid effectiveness indicator for unit tests.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信