{"title":"Speeding-Up Mutation Testing via Data Compression and State Infection","authors":"Qianqian Zhu, Annibale Panichella, A. Zaidman","doi":"10.1109/ICSTW.2017.25","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.25","url":null,"abstract":"Mutation testing is widely considered as a high-end test criterion due to the vast number of mutants it generates. Although many efforts have been made to reduce the computational cost of mutation testing, its scalability issue remains in practice. In this paper, we introduce a novel method to speed up mutation testing based on state infection information. In addition to filtering out uninfected test executions, we further select a subset of mutants and a subset of test cases to run leveraging data-compression techniques. In particular, we adopt Formal Concept Analysis (FCA) to group similar mutants together and then select test cases to cover these mutants. To evaluate our method, we conducted an experimental study on six open source Java projects. We used EvoSuite to automatically generate test cases and to collect mutation data. The initial results show that our method can reduce the execution time by 83.93% with only 0.257% loss in precision.","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"265 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124316413","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Hugo L. S. Araujo, Gustavo Carvalho, A. Sampaio, M. Mousavi, Masoumeh Taromirad
{"title":"A Process for Sound Conformance Testing of Cyber-Physical Systems","authors":"Hugo L. S. Araujo, Gustavo Carvalho, A. Sampaio, M. Mousavi, Masoumeh Taromirad","doi":"10.1109/ICSTW.2017.15","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.15","url":null,"abstract":"We present a process for sound conformance testing of cyber-physical systems, which involves functional but also non-functional aspects. The process starts with a hybrid model of cyber-physical systems in which the correct behavior of the system (at its interface level) is specified. Such a model captures both discrete behavior and evolution of continuous dynamics of the system in time. Since conformance testing inherently involves comparing continuous dynamics, the key parameters of the process are (1) the conformance bounds defining when two signals are sufficiently close to each other, and (2) the permitted error margin in the conformance analysis introduced by sampling of continuous signals. The final parameter of this process is (3) finding (and adjusting) the sampling rate of the dynamic behavior. In the specified process, we provide different alternatives for fixing the error margin of the conformance testing if the sampling rate is fixed, establishing the sampling rate if the error margin is fixed and finding conformance bounds once the sampling rate and the error margin are fixed.","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"8 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130692906","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jaganmohan Chandrasekaran, Huadong Feng, Yu Lei, D. Kuhn, R. Kacker
{"title":"Applying Combinatorial Testing to Data Mining Algorithms","authors":"Jaganmohan Chandrasekaran, Huadong Feng, Yu Lei, D. Kuhn, R. Kacker","doi":"10.1109/ICSTW.2017.46","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.46","url":null,"abstract":"Data mining algorithms are used to analyze and discover useful information from data. This paper presents an experiment that applies Combinatorial Testing (CT) to five data mining algorithms implemented in an open-source data mining software called WEKA. For each algorithm, we first run the algorithm with 51 datasets to study the impact different datasets have on the test coverage. We select one dataset that achieves the highest branch coverage. Next we construct positive and negative combinatorial test sets of configuration options and execute each test set with the selected dataset. Test effectiveness is measured using branch and mutation coverage. Our results suggest that when testing data mining algorithms: (1) larger datasets do not necessarily achieve higher coverage than smaller datasets, (2) test coverage increases progressively slower as test strength increases, and (3) branch coverage correlates well with mutation coverage.","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"52 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131243162","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Using Timed Base-Choice Coverage Criterion for Testing Industrial Control Software","authors":"Henning Bergstrom, Eduard Paul Enoiu","doi":"10.1109/ICSTW.2017.41","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.41","url":null,"abstract":"The base-choice criterion was proposed as a suitable technique for testing software based on its nominal choice of input parameters. Test cases are created based on this strategy by varying the values of one input parameter at a time while keeping the values of the other parameters fixed on the base choice. However, this strategy might not be as effective when used on industrial control software for testing timed behavior. We propose to incorporate time as another parameter when generating and executing tests by defining the timed base-choice coverage criterion. We performed an empirical evaluation using 11 industrial programs written in the IEC 61131-3 programming language. We found that tests generated for timed base-choice criterion show better code coverage (7% improvement) and fault detection (27% improvement) in terms of mutation score than tests satisfying base-choice coverage criterion. The results demonstrate the feasibility of applying timed base-choice criterion for testing industrial control software.","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125875483","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Chang Rao, Jin Guo, Nan Li, Yu Lei, Yadong Zhang, Yao Li, Yaxin Cao
{"title":"Applying Combinatorial Testing to High-Speed Railway Track Circuit Receiver","authors":"Chang Rao, Jin Guo, Nan Li, Yu Lei, Yadong Zhang, Yao Li, Yaxin Cao","doi":"10.1109/ICSTW.2017.39","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.39","url":null,"abstract":"A track circuit is a critical component of a modern railway system. The circuit is used to detect the absence of a train on rail tracks and produce corresponding control signals. A railway track has many block sections, each of which has a separate track circuit device. A track circuit receiver analyzes signals sent from a track circuit sender and produces a signal to control a relay. When the train is in a section, the relay must be down, otherwise, the relay must be up. We tested a track circuit receiver product that is widely used in high-speed railways in China. We extracted three input parameters from a signal sent to the receiver and create an Input Domain Model (IDM). We generated pair-wise test sets using the IDM. Moreover, from the requirements, we derived two UML finite state machines (FSMs) in the presence of multiple signals. In the FSMs, a transition can represent a signal input to the system. We first generated test paths from the FSMs, and then applied combinatorial testing to the parameters used in the test paths. We generated a total of 13,971 tests, which allowed us to detect four new faults in a release version of the product.","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"211 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122878051","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Macario Polo, Francisco Ruiz Romero, Rosana Rodriguez-Bobada Aranda, I. G. D. Guzmán
{"title":"Test Case Generation with Regular Expressions and Combinatorial Techniques","authors":"Macario Polo, Francisco Ruiz Romero, Rosana Rodriguez-Bobada Aranda, I. G. D. Guzmán","doi":"10.1109/ICSTW.2017.38","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.38","url":null,"abstract":"A test case describes a specific execution scenario of the system under test (SUT). Its goal is to discover errors by means of its oracle, that emits a pass or fail verdict depending on the SUT behavior. The test case has a sequence of calls to SUT's operations with specific test data, which may come from the application of a combinatorial algorithm. This paper describes a method to describe generic test scenarios by means of regular expressions, whose symbols point to a SUT operation. The tester assigns values to each operation's parameter. A further step expands the regular expression and produces a set of operation sequences, which are then passed to a combinatorial algorithm to generate actual test cases. Regular expressions are annotated with a set of when clauses, that are processed by the combinatorial algorithm to include the oracle in the generated test cases.","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"38 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131043695","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Combinatorial Testing on Implementations of HTML5 Support","authors":"Xi Deng, Tianyong Wu, Jun Yan, Jian Zhang","doi":"10.1109/ICSTW.2017.47","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.47","url":null,"abstract":"The fifth version of HTML standard, which is widely accepted in the diverse landscape of browser vendors and their continuously upgrading releases, requires web browsers to support playback of multimedia natively, instead of by third-party plugins. Works on testing browsers' implementations of the HTML5 support, however, are not efficient enough till now. Regarding a browser's processing on HTML5 element tags of web pages and attributes of multimedia files, this paper treats the browser as a parameterized black-box and employs the combinatorial testing technique to design web pages to test its system behavior. Experiments are conducted on two sets of browsers. The first set includes nine popular ones in both desktop and mobile for discovering the distribution of multimedia related errors, and the second set contains five versions of the same browser for revealing the evolution of such errors. The experimental results indicate that the proposed approach is promising as it can reveal errors in browsers by various vendors and in various versions, and that the upgrades of the browser will not necessarily fix the existing bugs, and even introduce new ones, due to inefficient testing.","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"195 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122850011","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Test Conglomeration - Proposal for Test Design Notation Like Class Diagram","authors":"Noriyuki Mizuno, Makoto Nakakuki, Yoshinori Seino","doi":"10.1109/ICSTW.2017.57","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.57","url":null,"abstract":"In recent years, software test has become huge and complicated and been required continuous change. In this situation, it is difficult to confirm overview of test and to share understanding with stakeholders. Furthermore, omissions of test case are often occurred because characteristics of traditional documents and difference in testing skills among test engineers. In this paper, \"Test Conglomeration\" notation that represents overview and consistent picture is proposed to deal with these problems.","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"29 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127720302","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Pattern-Based Usability Testing","authors":"F. Dias, A. C. Paiva","doi":"10.1109/ICSTW.2017.65","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.65","url":null,"abstract":"Usability is a critical aspect of software systems because poor user experience can lead users to choose other software. One way to improve usability is through testing. But, usability testing is a challenge because, most of the times, it can not be accomplished without the presence of real users, which is complex and requires a lot of effort. However, there are some aspects and usability guidelines that can be tested automatically. This paper presents a test approach that defines generic test strategies (test patterns) to test usability guidelines (or best practices). It is an extension to previous work on testing functional aspects of web applications through the GUI (PBGT – Pattern Based GUI Testing). The main goal of this work is to be the first step in extending PBGT's PARADIGM language with usability testing patterns, so that it is possible to build test models from which usability tests can be generated and automatically executed over a website. This paper presents a new usability test pattern, called \"Reachability Test Pattern\", which is validated in a case study performed over an academic software system available on the web.","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129809687","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Software Testing in Industry and Academia: A View of Both Sides in Japan","authors":"Satoshi Masuda","doi":"10.1109/ICSTW.2017.12","DOIUrl":"https://doi.org/10.1109/ICSTW.2017.12","url":null,"abstract":"Collaboration between industry and academia is important for solving problems and creating innovations. Both sides of industry and academia are important for each other, but sometimes they are incompatible. In this paper, we discuss software testing in industry and academia from their respective views in Japan on the basis of the author's experiences and studies. High quality software is required and its industry is influenced by economic circumstances. In Japan, there are also problems regarding cost reductions, production of high quality software, adapting businesses, and so on. We have made efforts to solve these problems by collaborating with people in both industry and academia. Our efforts include developing test methodologies, skill standards, education syllabi, and so on.","PeriodicalId":312017,"journal":{"name":"2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","volume":"9 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131268413","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}