Paolo Tell, Jacob B. Cholewa, P. Nellemann, M. Kuhrmann
{"title":"Beyond the spreadsheet: reflections on tool support for literature studies","authors":"Paolo Tell, Jacob B. Cholewa, P. Nellemann, M. Kuhrmann","doi":"10.1145/2915970.2916011","DOIUrl":"https://doi.org/10.1145/2915970.2916011","url":null,"abstract":"Background: Even though a number of tools are reported to be used by researchers undertaking systematic reviews, important shortages are still reported revealing how such solutions are unable to satisfy current needs. Method: Two research groups independently provided a design for a tool supporting systematic reviews. The resulting tools were assessed against the feature lists provided by prior research. Results: After presenting an overview of the tools and the core design decisions taken, we provide a feature analysis and a discussion regarding selected challenges deemed crucial to provide a proper tool support. Conclusions: Although the designed solutions do not yet support the entire systematic review process, their architecture has been designed to be flexible and extendable. After highlighting the difficulties of developing appropriate tools, we call for action: developing tools to support systematic reviews is a community project.","PeriodicalId":429555,"journal":{"name":"Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering","volume":"118 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132051468","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"An exploratory study on the effects of perceived value and stakeholder satisfaction on software projects","authors":"Hennie Huijgens, A. Deursen, R. Solingen","doi":"10.1145/2915970.2915987","DOIUrl":"https://doi.org/10.1145/2915970.2915987","url":null,"abstract":"Context: In this paper we present an exploratory study on the insights of organizations into the perceived value of their software projects. Our study is based on the notion that quantifying and qualifying project size, cost, duration and defects needs to be done in relation with stakeholder satisfaction and perceived value. Objectives: We expect that bringing perceived value into the equation will help in increasing the impact such organizations deliver. Method: In order to find out whether our approach is practically feasible in an industrial setting, we performed an exploratory study in a Belgian telecom company. Results: In this study we evaluate 22 software projects that were delivered during one release. Fifty-three (53) key stakeholders provide stakeholder satisfaction and perceived value measurements in 103 completed surveys. Conclusions: We conclude that a focus on shortening overall project duration, and improving communication on intermediate progress improved stakeholder satisfaction and perceived value. Our study does not provide any evidence that steering on costs helped to improve these.","PeriodicalId":429555,"journal":{"name":"Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering","volume":"17 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128242829","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
B. Olorisade, E. Quincey, P. Brereton, Péter András
{"title":"A critical analysis of studies that address the use of text mining for citation screening in systematic reviews","authors":"B. Olorisade, E. Quincey, P. Brereton, Péter András","doi":"10.1145/2915970.2915982","DOIUrl":"https://doi.org/10.1145/2915970.2915982","url":null,"abstract":"Background: Since the introduction of the systematic review process to Software Engineering in 2004, researchers have investigated a number of ways to mitigate the amount of effort and time taken to filter through large volumes of literature. Aim: This study aims to provide a critical analysis of text mining techniques used to support the citation screening stage of the systematic review process. Method: We critically re-reviewed papers included in a previous systematic review which addressed the use of text mining methods to support the screening of papers for inclusion in a review. The previous review did not provide a detailed analysis of the text mining methods used. We focus on the availability in the papers of information about the text mining methods employed, including the description and explanation of the methods, parameter settings, assessment of the appropriateness of their application given the size and dimensionality of the data used, performance on training, testing and validation data sets, and further information that may support the reproducibility of the included studies. Results: Support Vector Machines (SVM), Naïve Bayes (NB) and Committee of classifiers (Ensemble) are the most used classification algorithms. In all of the studies, features were represented with Bag-of-Words (BOW) using both binary features (28%) and term frequency (66%). Five studies experimented with n-grams with n between 2 and 4, but mostly the unigram was used. χ2, information gain and tf-idf were the most commonly used feature selection techniques. Feature extraction was rarely used although LDA and topic modelling were used. Recall, precision, F and AUC were the most used metrics and cross validation was also well used. More than half of the studies used a corpus size of below 1,000 documents for their experiments while corpus size for around 80% of the studies was 3,000 or fewer documents. The major common ground we found for comparing performance assessment based on independent replication of studies was the use of the same dataset but a sound performance comparison could not be established because the studies had little else in common. In most of the studies, insufficient information was reported to enable independent replication. The studies analysed generally did not include any discussion of the statistical appropriateness of the text mining method that they applied. In the case of applications of SVM, none of the studies report the number of support vectors that they found to indicate the complexity of the prediction engine that they use, making it impossible to judge the extent to which over-fitting might account for the good performance results. Conclusions: There is yet to be concrete evidence about the effectiveness of text mining algorithms regarding their use in the automation of citation screening in systematic reviews. The studies indicate that options are still being explored, but there is a need for better reporting as well as more exp","PeriodicalId":429555,"journal":{"name":"Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering","volume":"106 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128097866","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Eduardo Fernandes, Johnatan Oliveira, Gustavo Vale, T. Paiva, Eduardo Figueiredo
{"title":"A review-based comparative study of bad smell detection tools","authors":"Eduardo Fernandes, Johnatan Oliveira, Gustavo Vale, T. Paiva, Eduardo Figueiredo","doi":"10.1145/2915970.2915984","DOIUrl":"https://doi.org/10.1145/2915970.2915984","url":null,"abstract":"Bad smells are symptoms that something may be wrong in the system design or code. There are many bad smells defined in the literature and detecting them is far from trivial. Therefore, several tools have been proposed to automate bad smell detection aiming to improve software maintainability. However, we lack a detailed study for summarizing and comparing the wide range of available tools. In this paper, we first present the findings of a systematic literature review of bad smell detection tools. As results of this review, we found 84 tools; 29 of them available online for download. Altogether, these tools aim to detect 61 bad smells by relying on at least six different detection techniques. They also target different programming languages, such as Java, C, C++, and C#. Following up the systematic review, we present a comparative study of four detection tools with respect to two bad smells: Large Class and Long Method. This study relies on two software systems and three metrics for comparison: agreement, recall, and precision. Our findings support that tools provide redundant detection results for the same bad smell. Based on quantitative and qualitative data, we also discuss relevant usability issues and propose guidelines for developers of detection tools.","PeriodicalId":429555,"journal":{"name":"Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering","volume":"21 2","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114031150","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"The power of single and error annotations in category partition testing: an experimental evaluation","authors":"S. K. Khalsa, Y. Labiche, Johanna Nicoletta","doi":"10.1145/2915970.2915999","DOIUrl":"https://doi.org/10.1145/2915970.2915999","url":null,"abstract":"Category Partition (CP) is a black box testing technique that formalizes the specification of the input domain in a CP specification for the system under test. A CP specification is driven by tester's expertise and bundles parameters, categories (characteristics of parameters) and choices (acceptable values for categories) required for extensively testing the system. For completeness the choices correspond to permitted input values as well as some values to account for boundaries or robustness. These choices are then combined to form test frames on the basis of various criteria such as each choice or pairwise. To ensure that the combinations of choices are feasible and account for valid sets of user requirements, constraints are introduced to specify permitted combinations among choices, and to specify single or error choices. In a typical development environment where testing is driven by stringent deadlines a tester might have to decide how many constraints are enough to attain the maximum level of test completeness. The present work will assist a test engineer in making this decision. We conclude, on the basis of our experimental evaluation on academic and industrial case studies, that an equally effective test suite can be attained by meticulously defining error and single annotations in a CP specification while ignoring other constraints among choices.","PeriodicalId":429555,"journal":{"name":"Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering","volume":"10 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123957323","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Evaluating random mutant selection at class-level in projects with non-adequate test suites","authors":"Ali Parsai, Alessandro Murgia, S. Demeyer","doi":"10.1145/2915970.2915992","DOIUrl":"https://doi.org/10.1145/2915970.2915992","url":null,"abstract":"Mutation testing is a standard technique to evaluate the quality of a test suite. Due to its computationally intensive nature, many approaches have been proposed to make this technique feasible in real case scenarios. Among these approaches, uniform random mutant selection has been demonstrated to be simple and promising. However, works on this area analyze mutant samples at project level mainly on projects with adequate test suites. In this paper, we fill this lack of empirical validation by analyzing random mutant selection at class level on projects with non-adequate test suites. First, we show that uniform random mutant selection underachieves the expected results. Then, we propose a new approach named weighted random mutant selection which generates more representative mutant samples. Finally, we show that representative mutant samples are larger for projects with high test adequacy.","PeriodicalId":429555,"journal":{"name":"Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering","volume":"116 4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129088798","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Realistic assessment of software effort estimation models","authors":"Boyce Sigweni, M. Shepperd, Tommaso Turchi","doi":"10.1145/2915970.2916005","DOIUrl":"https://doi.org/10.1145/2915970.2916005","url":null,"abstract":"Context: It is unclear that current approaches to evaluating or comparing competing software cost or effort models give a realistic picture of how they would perform in actual use. Specifically, we're concerned that the usual practice of using all data with some holdout strategy is at variance with the reality of a data set growing as projects complete. Objective: This study investigates the impact of using unrealistic, though possibly convenient to the researchers, ways to compare models on commercial data sets. Our questions are does this lead to different conclusions in terms of the comparisons and if so, are the results biased e.g., more optimistic than those that might realistically be achieved in practice. Method: We compare a traditional approach based on leave one out cross-validation with growing the data set chronologically using the Finnish and Desharnais data sets. Results: Our realistic, time-based approach to validation is significantly more conservative than leave-one-out cross-validation (LOOCV) for both data sets. Conclusion: If we want our research to lead to actionable findings it's incumbent upon the researchers to evaluate their models in realistic ways. This means a departure from LOOCV techniques, while further investigation is needed for other validation techniques, such as k-fold validation.","PeriodicalId":429555,"journal":{"name":"Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering","volume":"367 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114528976","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Smitha Keertipati, Sherlock A. Licorish, Bastin Tony Roy Savarimuthu
{"title":"Exploring decision-making processes in Python","authors":"Smitha Keertipati, Sherlock A. Licorish, Bastin Tony Roy Savarimuthu","doi":"10.1145/2915970.2915993","DOIUrl":"https://doi.org/10.1145/2915970.2915993","url":null,"abstract":"The process by which norms are developed to become policies, the normative decision-making process, is not often explicit to stakeholders of Open Source Software (OSS) projects. Understanding the normative decision-making process is crucial for members if such projects are to evolve and succeed. In this paper, we investigated aspects of the normative decision-making processes of OSS development through the use of Python Enhancement Proposals (PEPs). We compared extracted process models with those that are advertised by the Python community to evaluate the extent to which those processes overlap. In addition, we assess members' involvement and contribution to these processes. Our work used structural and behavioral analysis techniques, and social network analysis metrics. We found that there were differences between the extracted processes and Python's advertised process, with the extracted processes being significantly more complex. These differences also extended to granular models used for managing social and technical aspects of the Python project. Furthermore, some key members were largely responsible for PEPs' success. Our extracted models could go a far way in helping the Python community to quickly understand decision-making processes in Python.","PeriodicalId":429555,"journal":{"name":"Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125805034","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Validating software measures using action research a method and industrial experiences","authors":"Vard Antinyan, M. Staron, A. Sandberg, J. Hansson","doi":"10.1145/2915970.2916001","DOIUrl":"https://doi.org/10.1145/2915970.2916001","url":null,"abstract":"Validating software measures for using them in practice is a challenging task. Usually more than one complementary validation methods are applied for rigorously validating software measures: Theoretical methods help with defining the measures with expected properties and empirical methods help with evaluating the predictive power of measures. Despite the variety of these methods there still remain cases when the validation of measures is difficult. Particularly when the response variables of interest are not accurately measurable and the practical context cannot be reduced to an experimental setup the abovementioned methods are not effective. In this paper we present a complementary empirical method for validating measures. The method relies on action research principles and is meant to be used in combination with theoretical validation methods. The industrial experiences documented in this paper show that in many practical cases the method is effective.","PeriodicalId":429555,"journal":{"name":"Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering","volume":"48 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128185069","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Episodic volunteering in open source communities","authors":"A. Barcomb","doi":"10.1145/2915970.2915972","DOIUrl":"https://doi.org/10.1145/2915970.2915972","url":null,"abstract":"Episodic volunteers, who prefer short term engagement to habitual contributions, are present in Free/Libre and Open Source Software (FLOSS) communities. Little is known about how they are viewed within their communities, how they view their communities, how community managers are managing them, or even how many episodic volunteers contribute to FLOSS projects and how much they contribute. Knowing more about the prevalence and management of episodic volunteers in FLOSS will help community managers make better decisions for engaging and utilizing these volunteers. My dissertation addresses these questions, providing a picture of what episodic volunteering looks like in the context of FLOSS communities.","PeriodicalId":429555,"journal":{"name":"Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering","volume":"30 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128248697","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}