Vikrant S. Kaulgud, A. Saxena, Sanjay Podder, V. Sharma, Chethana Dinakar
{"title":"Shifting Testing beyond the Deployment Boundary","authors":"Vikrant S. Kaulgud, A. Saxena, Sanjay Podder, V. Sharma, Chethana Dinakar","doi":"10.1145/2896941.2896954","DOIUrl":"https://doi.org/10.1145/2896941.2896954","url":null,"abstract":"Optimization of software testing has typically relied on a philosophy of 'shift left' where the focus is on using automation to start testing as early as possible in a software development life-cycle. With enterprises striving for continual business transformation, it becomes essential to focus on rapid and iterative software development. Moreover, understanding customer reactions, feedback and needs, and translating them as actionable insights to software development teams. This requires testing to 'shift right', i.e. move beyond the deployment boundary, and adopt a data-driven approach leveraging post-release production data and customer feedback. In this position paper we articulate the need for continuous, shift right testing, define a taxonomy for structuring testing types, analyses and data requirements. We end by providing a conceptual framework for applying shift right testing for Digital apps.","PeriodicalId":438234,"journal":{"name":"2016 IEEE/ACM International Workshop on Continuous Software Evolution and Delivery (CSED)","volume":"39 9","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-05-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114134832","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Cameron M. Hine, Jean-Guy Schneider, Jun Han, Steven Versteeg
{"title":"Enterprise Software Service Emulation: Constructing Large-Scale Testbeds","authors":"Cameron M. Hine, Jean-Guy Schneider, Jun Han, Steven Versteeg","doi":"10.1145/2896941.2896947","DOIUrl":"https://doi.org/10.1145/2896941.2896947","url":null,"abstract":"Constructing testbeds for systems which are interconnected with large networks of other software services is a challenging task. It is particularly difficult to create testbeds facilitating evaluation of the non-functional qualities of a system, such as scalability, that can be expected in production deployments. Software service emulation is an approach for creating such testbeds where service behaviour is defined by emulate-able models executed in an emulation runtime environment. We present (i) a meta-modelling framework supporting emulate-able service modelling (including messages, protocol, behaviour and states), and (ii) Kaluta, an emulation environment able to concurrently execute large numbers (thousands) of service models, providing a testbed which mimics the behaviour and characteristics of large networks of interconnected software services. Experiments show that Kaluta can emulate 10,000 servers using a single physical machine, and is a practical testbed for scalability testing of a real, enterprise-grade identity management suite. The insights gained into the tested enterprise system were used to enhance its design.","PeriodicalId":438234,"journal":{"name":"2016 IEEE/ACM International Workshop on Continuous Software Evolution and Delivery (CSED)","volume":"57 7 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-05-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121930508","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"History-Based Dynamic Test Case Prioritization for Requirement Properties in Regression Testing","authors":"Xiaolin Wang, Hong-wei Zeng","doi":"10.1145/2896941.2896949","DOIUrl":"https://doi.org/10.1145/2896941.2896949","url":null,"abstract":"Regression testing is an important but extremely costly and time-consuming process. Because of limited resources in practice, test case prioritization focuses on the improvement of testing efficiency. However, traditional test case prioritization techniques emphasize only one-time testing without considering huge historical data generated in regression testing. This paper proposes an approach to prioritizing test cases based on historical data. Requirements are a significant factor in the testing process, the priorities of test cases are initialized based on requirement priorities in our history-based approach, and then are calculated dynamically according to historical data in regression testing. To evaluate our approach, an empirical study on an industrial system is conducted. Experimental results show an improved performance for our proposed method using measurements of Average Percentage of Faults Detected and Fault Detection Rate.","PeriodicalId":438234,"journal":{"name":"2016 IEEE/ACM International Workshop on Continuous Software Evolution and Delivery (CSED)","volume":"413 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-05-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132139140","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Who Needs Release and DevOps Engineers, and Why?","authors":"Noureddine Kerzazi, Bram Adams","doi":"10.1145/2896941.2896957","DOIUrl":"https://doi.org/10.1145/2896941.2896957","url":null,"abstract":"The recent surge in interest in continuous delivery has opened up the job market for release and DevOps engineers. However, despite an increasing number of conferences and publications on continuous delivery, smaller companies and start-ups still have a hard time determining the core tasks their future release and DevOps engineers should be responsible for (and what the differences between those two roles are), while universities are not sure what essential techniques and skills they should teach to their students. This paper performs an empirical analysis of online job postings to determine and compare the main tasks of release and DevOps engineers, globally and across countries. Our qualitative analysis shows that automation is the most important activity across the three roles, as articulated in job posting description data, and that the release engineer role combines the top activities of the DevOps and more traditional build engineer roles. Finally, different countries have a moderate degree of similarity between their ads, although each country has its specific focus.","PeriodicalId":438234,"journal":{"name":"2016 IEEE/ACM International Workshop on Continuous Software Evolution and Delivery (CSED)","volume":"12 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-05-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128576286","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Trace-Based Test Selection to Support Continuous Integration in the Automotive Industry","authors":"Sebastian Vöst, S. Wagner","doi":"10.1145/2896941.2896951","DOIUrl":"https://doi.org/10.1145/2896941.2896951","url":null,"abstract":"System testing in the automotive industry is a very expensive and time-consuming task of growing importance, because embedded systems in the domain are distributed over numerous controllers (ECUs). Modern software development techniques such as continuous integration require regular, repeated and fast testing. To achieve this in the automotive domain, test suites for a specific software change must be tailored.We propose a novel test selection technique for system-level functions in the automotive industry based on component and communication models. The idea is to follow input and output signals that are used in the testing steps through the ECUs implementing a function. We select only those tests for a planned integration in which at least one of the signals sent in its steps is processed by the ECU that was changed and thus triggered the integration. The technique is well-suited for black-box testing since it requires only the full test suite specification and the system architecture.We applied the technique to a test suite of the Active Cruise Control function at BMW Group in the context of hardware-in-the-loop system testing and found the possible reduction rates to be 82.3% on average in comparison to the full test suite.Possible future work includes the evaluation with a wider set of functions, the evaluation of the fault detection rate, further automation and combination with other test selection techniques.","PeriodicalId":438234,"journal":{"name":"2016 IEEE/ACM International Workshop on Continuous Software Evolution and Delivery (CSED)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-05-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125888189","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}