Brazilian Symposium on Systematic and Automated Software Testing最新文献

{"title":"Search-Based Higher Order Mutation Testing: A Mapping Study","authors":"Jackson A. Prado Lima, S. Vergilio","doi":"10.1145/3266003.3266013","DOIUrl":"https://doi.org/10.1145/3266003.3266013","url":null,"abstract":"Higher Order Mutation Testing (HOMT) uses mutants derived by applying mutation operators more than once in the program under test. This kind of test increases efficacy, but can be more expensive and lead to some challenges such as the large space of mutants. The challenges can be managed in the Search-Based Software Engineering field to seek only interesting mutants that satisfy some desired conditions regarding, for example, type of revealed faults or reduction of equivalent mutants. For this reason, we observe a growing interest in search-based HOMT and an increasing number of related studies. To better characterize such studies, this paper presents results of a mapping on search-based HOMT. We found 25 primary studies on this subject and identified the most common search-based algorithms, preferred fitness functions, addressed programming languages, and evaluation aspects. In addition to this, some trends and research opportunities are identified that allow researchers to direct future investigations.","PeriodicalId":208536,"journal":{"name":"Brazilian Symposium on Systematic and Automated Software Testing","volume":"81 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121184965","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Study on the Impact of Model Evolution in MBT Suites","authors":"Anderson G. F. Silva, W. Andrade, Everton L. G. Alves","doi":"10.1145/3266003.3266009","DOIUrl":"https://doi.org/10.1145/3266003.3266009","url":null,"abstract":"Software testing is known to be a key-factor for the success of a software development project. In this context, Model-Based Testing (MBT) plays an important role by providing an automated way for generating test cases from system models. However, although MBT can be helpful for creating sound test cases, its use can be very sensitive to model changes. Model edits often lead to a great number of obsolete test cases, as the software and its models evolve. This fact is even more evident in agile projects where requirement artifacts are very volatile. This paper presents an empirical study designed for investigating how model edits can impact MBT test suites. For that, we run a case study in the context of two industrial projects that apply agile methodologies combined with the use of MBT. We observed the evolution of specification models and their impact on generated MBT suites. Our results showed that 86% of the generated test cases were discarded due to model edits and their impact. However, a deeper analysis found that 52% of these tests were impacted only by syntactic model edits, which indicate they could be reused with little revision efforts.","PeriodicalId":208536,"journal":{"name":"Brazilian Symposium on Systematic and Automated Software Testing","volume":"43 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121361847","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Using a Search and Model Based Framework to Improve Robustness Tests in Cloud Platforms","authors":"W. Cardoso, E. Martins","doi":"10.1145/3266003.3266011","DOIUrl":"https://doi.org/10.1145/3266003.3266011","url":null,"abstract":"Cloud computing has changed the way IT companies use and provide their services. Due to the elasticity in such infrastructures, the financial economic becomes attractive in different scenarios, from small to large business. The term cloud computing refers to software and hardware delivered as a service, and the systems that control the hardware in data centers.\u0000 Test the cloud infrastructure is challenging because resources appear to be infinite. On the one hand, a system scale quickly, from 1 server to 1,000 servers in seconds. On the other, if a failure occurs, it is difficult to reproduce and debug. It is common in such cases the experient testing team writing down most of their tests, which although effective to reveal bugs is expensive and error-prone in practice. Further, most cloud software programs are required to stay up all the time, which need them to implement some failure tolerant mechanisms. Test these systems concerning only their functionalities is not enough to reveal robustness flaws as functional testing is not aimed to put the system in anomalous conditions.\u0000 Cloud robustness testing approaches lack in considering large deployments due to the difficulty to instantiate them up, thereby most of these scenarios are ignored. But, in practice, the more severe failures occur in large deployments in tricky scenarios. Our study is aimed at improving tests by generating behavioral models from the testing specification and robustness tests from the models.\u0000 This paper presents a method for robustness testing of a cloud platform. We started with OpenStack, a cloud software that counts with components to manage identities, images, instances, networks, storages, etc. Our method is supported by a tool suite called StateMutest, which generate test cases from UML state models, among other capabilities. The method comprises the robustness behavior modeling, proceeding with the search-based approach for test case generation.\u0000 We compared the results obtained with those provided by OpenStack community. Results show the effectiveness of the proposed method, as it improves on results obtained by the community.","PeriodicalId":208536,"journal":{"name":"Brazilian Symposium on Systematic and Automated Software Testing","volume":"48 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116980871","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Testing Strategies for Smart Cities applications: A Systematic Mapping Study","authors":"Alex Costa, Leopoldo Teixeira","doi":"10.1145/3266003.3266005","DOIUrl":"https://doi.org/10.1145/3266003.3266005","url":null,"abstract":"Context: Smart Cities are urban areas that enable the development of applications to improve city resources management, through the use of information technology such as Internet of Things and cloud computing, as well as government data availability and citizen participation. Some challenges identified in the development of solutions for this context are: scalability, modularity and security. Goal: Testing activities are critical to the verification and validation of Smart City solutions, so our goal is to develop a map of test strategies for applications developed in the context of Smart Cities. Method: For this study we defined a systematic literature review protocol to identify, select, analyze and synthesize the results of previously published empirical studies in the software engineering literature, related to testing Smart Cities applications. Results: In this systematic mapping, 13 articles were selected, that have identified test strategies used by developers for the application testing process. We have also identified some difficulties faced in the process of testing these applications through reports present in the works selected in this mapping. Conclusion: Our research synthesized evidence that we hope might serve as a data source for academic research and industrial practice. As future work we plan to apply our results highlighting a case study in real applications to validate the collected evidence.","PeriodicalId":208536,"journal":{"name":"Brazilian Symposium on Systematic and Automated Software Testing","volume":"14 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122520565","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Evaluating the Impact of Different Testers on Model-based Testing","authors":"Henrique Neves da Silva, Guilherme Ricken Mattiello, A. T. Endo, É. F. D. Souza, S. Souza","doi":"10.1145/3266003.3266012","DOIUrl":"https://doi.org/10.1145/3266003.3266012","url":null,"abstract":"Context: Model-Based Testing (MBT) is an approach that allows testers to represent the behavior of the system under test as models, specifying inputs and their expected outputs. From such models, existing tools might be employed to generate test cases automatically. While MBT represents a promising step towards the automation of test case generation, the quality of the model designed by the tester may impact, either positively or negatively, on its ability to reveal faults (i.e., the test effectiveness). Objective: In this context, we conducted a preliminary experiment to evaluate the impact caused by different testers when designing a test model for the same functionality. Method: In the experiment, the participants used Event Sequence Graphs and its supporting tool FourMA to create test models for two mobile apps: arXiv-mobile and WhoHasMyStuff. From the test models, test cases were generated using FourMA and concretized by means of the Robotium framework. In order to measure the impact of different testers, we employed code coverage (namely, instruction and branch coverage) as an estimation of test effectiveness. Results: Based on the results obtained, we observe high variation of code coverage among the testers. No tester was capable of producing a test model that subsumes all other testers' models with respect to code coverage. Moreover, factor learning seems not to reduce the code coverage variation. The relation between model size, modeling time, and code coverage were inconclusive. Conclusion: We conclude that further research effort on the MBT's modeling step is required to not only reduce the variation between testers, but also improving its effectiveness.","PeriodicalId":208536,"journal":{"name":"Brazilian Symposium on Systematic and Automated Software Testing","volume":"3 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121531296","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Assessing Agile Testing Practices for Enterprise Systems: A Survey Approach","authors":"Rafael Fazzolino, Henrique Medrado de Faria, L. Amaral, E. Canedo, G. Rodrigues, R. Bonifácio","doi":"10.1145/3266003.3266010","DOIUrl":"https://doi.org/10.1145/3266003.3266010","url":null,"abstract":"Testing enterprise systems is a challenging task. In this context, besides considering user requirements and business rules, there are several concerns that must also be considered when designing a test suite, including user interaction, integration with legacy systems, and access to relational or non-relational databases. In addition, test engineers often have to decide whether to test in the different levels (e.g., unit, integration, and system testing) or not, whether to use mock objects or not, and so on. Sadly, these decisions often conflict to each other. In this paper, we report an 18-month experience on using different approaches and tools to test two enterprise systems (SISBOL and SISDOT) from the Brazilian Army. SISBOL is an enterprise system based on the REST architectural style, using JEE (Java Enterprise Edition) in the business and integration tiers and AngularJS in the presentation tier. On the other hand, SISDOT is a \"pure\" Java enterprise system, using Java Server Faces on the presentation tier. We found that using executable specifications (as recommended in approaches based on Behavior Driven Development) is not only valid in test first scenarios as well as giving confidence on the system requirements validation, but they also help to identify new execution paths when the source code had been already implemented---suggesting that it is also promising to introduce a BDD approach in later stages of the development than testing pure JEE systems at the integration level.","PeriodicalId":208536,"journal":{"name":"Brazilian Symposium on Systematic and Automated Software Testing","volume":"273 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133567570","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Gamification in Software Testing: A Characterization Study","authors":"G. Jesus, F. Ferrari, D. Porto, S. Fabbri","doi":"10.1145/3266003.3266007","DOIUrl":"https://doi.org/10.1145/3266003.3266007","url":null,"abstract":"Context: Testing is a fundamental activity in the software development cycle. Revealing software faults is its main objective. Despite that, testing is considered unpleasant, dull and tedious. As a result, there is a lack of expertise among professionals while many projects fail. Gamification is a promising way to address testing issues; it is a new trend being used mostly to increase motivation, engagement and performance with the use of game elements in non-game contexts. Objective: To describe results of a study that aimed to characterize how gamification has been explored to support software testing. Method: The studies that compose our baseline for analysis and discussion were obtained through a systematic mapping carried out following a research protocol. To retrieve relevant literature, we applied automatic search and backward snowballing. At the end, we selected 15 studies that we analyzed and classified according to six perspectives: application context, used gamifcation elements, gamification goals, testing techniques, testing levels, and testing process phases. Results: The most used gamification elements are points, leader boards, and levels, and unit testing and functional testing are the level and technique most addressed in the studies, respectively. Conclusion: Gamification is a rising research topic, especially in the software testing field. The increasing interest for gamification has the potential do lead to positive outcomes. The map presented in this paper can be a useful resource for the identification of gaps and for triggering new research initiatives.","PeriodicalId":208536,"journal":{"name":"Brazilian Symposium on Systematic and Automated Software Testing","volume":"93 4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122536972","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Mutation Operators for Code Annotations","authors":"Pedro Pinheiro, J. C. Viana, Leo Fernandes, Márcio Ribeiro, F. Ferrari, B. Neto, Rohit Gheyi","doi":"10.1145/3266003.3266006","DOIUrl":"https://doi.org/10.1145/3266003.3266006","url":null,"abstract":"Mutation testing is a technique that injects artificial faults to check whether the existing test suite can detect them. To inject the faults, mutation testing tools rely on mutation operators, such as Arithmetic Operator Replacement, Relational Operator Replacement, Statement Deletion, and Overriding Method Rename. Most of the proposed mutation operators have focused on imperative language constructs and object-oriented constructs. However, many projects make use of meta-programming through code annotations, and mutation operators to simulate annotation-related faults are missing. To minimize this problem, in this paper we propose a set of mutation operators capable of simulating annotation-related faults. In particular, we define nine operators. To evaluate our operators, we mined Java projects that make heavy use of annotations to identify annotation-related faults. We analyzed 100 faults and categorized them as \"misuse\", when the developer does not know how to use the annotations in a proper way; and \"wrong annotation parsing\", when the developer wrongly parses annotation code (by using refection, for example). Our operators are able to simulate 95 out of the 100 mined faults. In particular, three operators are able to simulate up to 82% of the analyzed faults. As implications for practice, our operators can help developers to improve their test suite and the code responsible for parsing annotated code. Also, they can help developers to avoid annotation-related faults.","PeriodicalId":208536,"journal":{"name":"Brazilian Symposium on Systematic and Automated Software Testing","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122913311","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Test Framework for Jenkins Shared Libraries","authors":"Renan Vieira Antunes, Gabriela Matias Navarro, Simone Hanazumi","doi":"10.1145/3266003.3266008","DOIUrl":"https://doi.org/10.1145/3266003.3266008","url":null,"abstract":"Delivering new features to users as fast and efficiently as possible is a common challenge on software development. A continuous delivery and integration pipeline can be used for this purpose, since it provides an automated way to orchestrate a series of steps for building, versioning, testing and deploying an application code.\u0000 The newest version of Jenkins works with a concept called pipeline-as-code, where instead of configuring jobs on Jenkins to build, deploy or test the application, the user can implement a shared library with the same purpose. As a result, Jenkins job settings have been transformed into Groovy code, which enables the development of automated unit tests. However, using this pipeline brings the challenge of guaranteeing the high quality of this orchestration process. Unit tests by themselves are not enough to make sure that the implementation of new pipeline features did not add new bugs or changed the pipeline behavior. To do this validation we need to write functional tests, but that requires understanding of Jenkins and Groovy which can make this task complex. For this reason, a test framework was created as a shared library to facilitate the creation of functional tests for Jenkins shared libraries. The framework can execute several pipeline scenarios described by the user and apply them to different projects. The framework helps to ensure a high quality in the pipeline code used to ship your application from development environment until production, resulting in a more reliable process.","PeriodicalId":208536,"journal":{"name":"Brazilian Symposium on Systematic and Automated Software Testing","volume":"53 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114487216","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Multi-objective optimization approach for selection of second order mutant generation strategies","authors":"Jackson A. Prado Lima, S. Vergilio","doi":"10.1145/3128473.3128479","DOIUrl":"https://doi.org/10.1145/3128473.3128479","url":null,"abstract":"The use of Higher-Order Mutants (HOMs) presents some advantages concerning the traditional use of First-Order Mutants (FOMs). HOMs can better simulate real and subtle faults, reduce the number of generated mutants and test cases, and so on. However, the HOM space is potentially huge, and an efficient strategy to generate the best HOMs is fundamental. In the literature different strategies were proposed and evaluated, mainly to generate Second-Order Mutants (SOMs), but none has been proved to perform better in different situations. Due to this, the selection of the best strategy is an important task. Most times a lot of experiments need to be conducted. To help the tester in this task and to allow the use of HOMs in practice, this paper proposes a hyper-heuristic approach. Such approach is based on NSGA-II and uses the selection method Choice Function to automatically choose among different Low-Level Heuristics (LLHs), which, in this case, are search-operators related to existing SOM generation strategies. The performance of each LLH is related to some objectives such as the number of SOMs generated, the capacity to capture subtler faults and replace the constituent FOMs. In comparison with existing strategies, our approach obtained better results considering the used objectives, and statistically equivalent results considering mutation score with respect to the FOMs.","PeriodicalId":208536,"journal":{"name":"Brazilian Symposium on Systematic and Automated Software Testing","volume":"34 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132306173","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}