Journal of Systems and Software最新文献

{"title":"An empirical investigation into the capabilities of anomaly detection approaches for test smell detection","authors":"Valeria Pontillo ,&nbsp;Luana Martins ,&nbsp;Ivan Machado ,&nbsp;Fabio Palomba ,&nbsp;Filomena Ferrucci","doi":"10.1016/j.jss.2024.112320","DOIUrl":"10.1016/j.jss.2024.112320","url":null,"abstract":"<div><div>Test smells are symptoms of sub-optimal design choices adopted when developing test cases. Previous research has demonstrated their harmfulness for test code maintainability and effectiveness, showing their impact on test code quality. As such, the quality of test cases affected by test smells is likely to deviate significantly from the quality of test cases not affected by any smell and might be classified as <em>anomalies</em>. In this paper, we challenge this observation by experimenting with three anomaly detection approaches based on machine learning, cluster analysis, and statistics to understand their effectiveness for the detection of four test smells, i.e., <em>Eager Test</em>, <em>Mystery Guest</em>, <em>Resource Optimism</em>, and <em>Test Redundancy</em> on 66 open-source <span>Java</span> projects. In addition, we compare our results with state-of-the-art heuristic-based and machine learning-based baselines. Our ultimate goal is not to prove that anomaly detection methods are better than existing approaches, but to objectively assess their effectiveness in this domain. The key findings of the study show that the <em>F-Measure</em> of anomaly detectors never exceeds 47%, obtained in the <em>Eager Test</em> detection using the statistical approach, while the <em>Recall</em> is generally higher for the statistical and clustering approaches. Nevertheless, the anomaly detection approaches have a higher <em>Recall</em> than the heuristic and machine learning-based techniques for all test smells. The low <em>F-Measure</em> values we observed for anomaly detectors provide valuable insights into the current limitations of anomaly detection in this context. We conclude our study by elaborating on and discussing the reasons behind these negative results through qualitative investigations. Our analysis shows that the detection of test smells could depend on the approach exploited, suggesting the feasibility of developing a meta-approach.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"222 ","pages":"Article 112320"},"PeriodicalIF":3.7,"publicationDate":"2024-12-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143104092","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Model-driven engineering for Software Architecture","authors":"Alessio Bucaioni,&nbsp;Amleto Di Salle,&nbsp;Ludovico Iovino,&nbsp;Peng Liang","doi":"10.1016/j.jss.2024.112321","DOIUrl":"10.1016/j.jss.2024.112321","url":null,"abstract":"","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"223 ","pages":"Article 112321"},"PeriodicalIF":3.7,"publicationDate":"2024-12-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143463320","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Characterising reproducibility debt in scientific software: A systematic literature review","authors":"Zara Hassan ,&nbsp;Christoph Treude ,&nbsp;Michael Norrish ,&nbsp;Graham Williams ,&nbsp;Alex Potanin","doi":"10.1016/j.jss.2024.112327","DOIUrl":"10.1016/j.jss.2024.112327","url":null,"abstract":"<div><h3>Context:</h3><div>In scientific software, the inability to reproduce results is often due to technical issues and challenges in recreating the full computational workflow from the original analysis. We conceptualise this problem as <em>Reproducibility Debt</em> (RpD). Much research has been performed to propose solutions to tackle these issues across various computational science disciplines. It is essential to identify and accumulate existing knowledge on reproducibility issues and state-of-the-art solutions so as to provide researchers and practitioners with information that enables further research activities and RpD management in practice.</div></div><div><h3>Objective:</h3><div>In the context of scientific software, we aim to characterise RpD by providing a taxonomy of issues contributing towards its emergence and identification (causes, effects) and the common solutions discussed in the existing literature.</div></div><div><h3>Method:</h3><div>We conducted a systematic literature review, considering 2198 studies until January 2024, including 214 primary studies.</div></div><div><h3>Results:</h3><div>We propose the first taxonomy of RpD items consisting of 37 causes attributed towards its emergence, 63 corresponding effects under seven main categories, and 29 prevention strategies. We also identify 39 specialised tools/frameworks supporting reproducibility.</div></div><div><h3>Conclusion:</h3><div>The main contributions of this work are (1) a formal definition of RpD; (2) a taxonomy of issues contributing towards RpD; (3) a list of causes and effects having implications for software professionals to identify and measure RpD in their projects; (4) a list of strategies and tools to prevent or remove RpD; (5) the identification of gaps in existing research to guide future studies.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"222 ","pages":"Article 112327"},"PeriodicalIF":3.7,"publicationDate":"2024-12-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143104093","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Effectiveness of symmetric metamorphic relations on validating the stability of code generation LLM","authors":"Pak Yuen Patrick Chan ,&nbsp;Jacky Keung ,&nbsp;Zhen Yang","doi":"10.1016/j.jss.2024.112330","DOIUrl":"10.1016/j.jss.2024.112330","url":null,"abstract":"<div><div>Pre-trained large language models (LLMs) are increasingly used in software development for code generation, with a preference for private LLMs over public ones to avoid the risk of exposing corporate secrets. Validating the stability of these LLMs’ outputs is crucial, and our study proposes using symmetric Metamorphic Relations (MRs) from Metamorphic Testing (MT) for this purpose. Our study involved an empirical experiment with ten LLMs (eight private and two public) and two publicly available datasets. We defined seven symmetric MRs to generate “Follow-up” datasets from “Source” datasets for testing. Our evaluation aimed to detect violations (inconsistent predictions) between “Source” and “Follow-up” datasets and assess the effectiveness of MRs in identifying correct and incorrect non-violated predictions from ground truths. Results showed that one public and four private LLMs did not violate “Case transformation of prompts” MR. Furthermore, effectiveness and performance results indicated that proposed MRs are effective tools for explaining the instability of LLM's outputs by “Case transformation of prompts”, “Duplication of prompts”, and “Paraphrasing of prompts”. The study underscored the importance of enhancing LLMs’ semantic understanding of prompts for better stability and highlighted potential future research directions, including exploring different MRs, enhancing semantic understanding, and applying symmetry to prompt engineering.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"222 ","pages":"Article 112330"},"PeriodicalIF":3.7,"publicationDate":"2024-12-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143104094","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Analysing Self-Adaptive Systems as Software Product Lines","authors":"Juliane Päßler ,&nbsp;Maurice H. ter Beek ,&nbsp;Ferruccio Damiani ,&nbsp;Einar Broch Johnsen ,&nbsp;S. Lizeth Tapia Tarifa","doi":"10.1016/j.jss.2024.112324","DOIUrl":"10.1016/j.jss.2024.112324","url":null,"abstract":"<div><div>Self-adaptation is a crucial feature of autonomous systems that must cope with uncertainties in, e.g., their environment and their internal state. Self-adaptive systems (SASs) can be realised as two-layered systems, introducing a separation of concerns between the domain-specific functionalities of the system (the <em>managed</em> subsystem) and the adaptation logic (the <em>managing</em> subsystem), i.e., introducing an external feedback loop for managing adaptation in the system. We present an approach to model SASs as dynamic software product lines (SPLs) and leverage existing approaches to SPL-based analysis for the analysis of SASs. To do so, the functionalities of the SAS are modelled in a feature model, capturing the SAS’s variability. This allows us to model the managed subsystem of the SAS as a family of systems, where each family member corresponds to a valid feature configuration of the SAS. Thus, the managed subsystem of an SAS is modelled as an SPL model; more precisely, a probabilistic featured transition system. The managing subsystem of an SAS is modelled as a control layer capable of dynamically switching between these valid configurations, depending on both environmental and internal conditions. We demonstrate the approach on a small-scale evaluation of a self-adaptive autonomous underwater vehicle used for pipeline inspection, which we model and analyse with the feature-aware probabilistic model checker ProFeat. The approach allows us to analyse probabilistic reward and safety properties for the SAS, as well as the correctness of its adaptation logic.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"222 ","pages":"Article 112324"},"PeriodicalIF":3.7,"publicationDate":"2024-12-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143104089","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Techniques for eliciting IoT requirements: Sensorina Map and Mind IoT","authors":"Sabrina Souza ,&nbsp;Eriky Rodrigues ,&nbsp;Maria Meireles ,&nbsp;Tanara Lauschner ,&nbsp;Leandro Carvalho ,&nbsp;José Carlos Maldonado ,&nbsp;Tayana Conte","doi":"10.1016/j.jss.2024.112323","DOIUrl":"10.1016/j.jss.2024.112323","url":null,"abstract":"<div><h3>Context:</h3><div>The Internet of Things (IoT) involves heterogeneous devices that interact and process data via the Internet. In the development of IoT systems, requirement elicitation is crucial. However, challenges such as heterogeneity, interoperability, scalability, and requirements volatility necessitate new approaches or adapting traditional techniques.</div></div><div><h3>Objective:</h3><div>In this context, this work proposes the Sensorina Map and IoT Mind as techniques adapted from the Empathy Map and Mind Map, respectively, to support requirement elicitation in IoT systems.</div></div><div><h3>Method:</h3><div>Two empirical studies were conducted in an academic environment to assess the feasibility of the techniques, then, a case study in industry environment.</div></div><div><h3>Results:</h3><div>The first study analyzed the ease of use and evaluated if it assisted software engineers in remembering the system requirements. The participants’ perceptions were collected through a Focus Group, refining the techniques. Subsequently, an observational study evaluated the techniques’ usefulness and ease of use. The results of the study demonstrated that the participants considered the methods feasible. The case study results revealed that the Sensorina Map is more suitable for advanced stages. At the same time, the Mind IoT suits better the initial phases, emphasizing the need for practical examples and adaptations to suit diverse user profiles.</div></div><div><h3>Conclusion:</h3><div>This work is expected to advance research in IoT systems and benefit professionals and researchers in this area.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"222 ","pages":"Article 112323"},"PeriodicalIF":3.7,"publicationDate":"2024-12-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143104097","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"COMET: Generating commit messages using delta graph context representation","authors":"Abhinav Reddy Mandli ,&nbsp;Saurabhsingh Rajput ,&nbsp;Tushar Sharma","doi":"10.1016/j.jss.2024.112307","DOIUrl":"10.1016/j.jss.2024.112307","url":null,"abstract":"<div><div>Commit messages explain code changes in a commit and facilitate collaboration among developers. Several commit message generation approaches have been proposed; however, they exhibit limited success in capturing the context of code changes. We propose <strong><span>Comet</span> (<u>C</u>ontext-Aware C<u>o</u>mmit <u>Me</u>ssage Genera<u>t</u>ion)</strong>, a novel approach that captures context of code changes using a graph-based representation and leverages a transformer-based model to generate high-quality commit messages. Our proposed method utilizes <em>delta graph</em> that we developed to effectively represent code differences. We also introduce a customizable quality assurance module to identify optimal messages, mitigating subjectivity in commit messages. Experiments show that <span>Comet</span> outperforms state-of-the-art techniques in terms of <span>bleu</span>-norm and <span>meteor</span> metrics while being comparable in terms of <span>rouge-l</span>. Additionally, we compare the proposed approach with the popular <span>gpt-3.5-turbo</span> model, along with <span>gpt-4</span>—the most capable GPT model, over zero-shot, one-shot, and multi-shot settings. We found <span>Comet</span> outperforming the GPT models, on five and four metrics respectively and provide competitive results with the two other metrics. The study has implications for researchers, tool developers, and software developers. Software developers may utilize <span>Comet</span> to generate context-aware commit messages. Researchers and tool developers can apply the proposed <em>delta graph</em> technique in similar contexts, like code review summarization.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"222 ","pages":"Article 112307"},"PeriodicalIF":3.7,"publicationDate":"2024-12-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143171459","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"MITU: Locating relevant tutorial fragments of APIs with multi-source API knowledge","authors":"Di Wu ,&nbsp;Hongyu Zhang ,&nbsp;Yang Feng ,&nbsp;Zhenjiang Dong","doi":"10.1016/j.jss.2024.112296","DOIUrl":"10.1016/j.jss.2024.112296","url":null,"abstract":"<div><div>API tutorials are vital resources as they can help developers learn how to use the APIs. An API tutorial is usually split into a number of consecutive units that describe the same topic, denoted as <em>tutorial fragments</em>. We treat a tutorial fragment explaining how to use an API as a <em>relevant</em> fragment of the API. Locating relevant tutorial fragments of an API can help developers understand and learn APIs. Existing approaches often train location models using API knowledge from a single resource (e.g., API tutorials). In practice, API knowledge from multiple resources such as API tutorials, Stack Overflow (SO) posts, and API specifications (denoted as <em>multi-source API knowledge</em>) is available to help locate relevant fragments of APIs. While leveraging multi-source API knowledge is intuitively more beneficial, it is a challenging task to use multi-source API knowledge due to <em>diverse distribution</em> and <em>imbalanced distribution</em> issues. Here, the diverse distribution denotes that the data in the same resource are close to each other in the feature space, while data in different resources are far away from each other. The imbalanced distribution denotes that the amount of relevant data is less than the amount of irrelevant data. In this paper, we propose a novel approach called MITU (using <u><strong>M</strong></u>ulti-source AP<u><strong>I</strong></u> knowledge to locate relevant <u><strong>TU</strong></u>torial fragments) to alleviate these two challenges. For the diverse distribution problem, MITU can project multi-source API knowledge to a correlated space where their distributions become similar. For the imbalanced distribution problem, MITU can minimize the misclassification cost when learning multi-source API knowledge. More specifically, we first collect multi-source API knowledge from API specifications, SO posts, and API tutorials, respectively. Then, we train a cost-sensitive subspace analysis based location model, which can make full use of multi-source API knowledge by addressing issues of diverse and imbalanced distributions. At last, relevant tutorial fragments of APIs can be located by consulting the trained model. We evaluate MITU on Java and Android multi-source API knowledge datasets containing a total of 44,064 samples. Experimental results show that MITU is effective and outperforms the existing approaches. Moreover, our user study confirms the effectiveness of MITU in practice.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"222 ","pages":"Article 112296"},"PeriodicalIF":3.7,"publicationDate":"2024-12-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143104098","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A structural taxonomy for lifted software product line analyses","authors":"Logan Murphy,&nbsp;Mahmood Saifi,&nbsp;Alessio Di Sandro,&nbsp;Marsha Chechik","doi":"10.1016/j.jss.2024.112280","DOIUrl":"10.1016/j.jss.2024.112280","url":null,"abstract":"<div><div>A software product line (SPL) is a structured collection of distinct software products developed from a common set of artifacts. SPLs can encompass millions of products, so analysing each product in a brute-force manner is infeasible. To analyse SPLs directly, analyses must be <em>lifted</em>, i.e., redefined to accommodate the semantics of SPLs. Over the past two decades, many kinds of analyses have been lifted from products to SPLs. Looking at the landscape of lifted analyses, we observe various <em>techniques</em> for lifting which vary across numerous dimensions. To help engineers and research navigate this landscape, we propose a classification scheme for lifted analyses based on a set of features lifted analyses can exhibit. We then conduct a systematic literature review (SLR) analysing the landscape of lifted analyses produced over the last 20 years. We analyse 140 research papers which discuss the design and implementation of lifted analyses. We provide quantitative analysis of the types of analyses which have been lifted, and apply our taxonomy to clarify <em>how</em> lifting was accomplished. We discuss examples of how each of the lifting methods have been applied, and identify gaps in the research literature which may provide directions for future work.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"222 ","pages":"Article 112280"},"PeriodicalIF":3.7,"publicationDate":"2024-12-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143171458","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"STILE: A tool for optimizing E2E web test scripts parallelization","authors":"Dario Olianas ,&nbsp;Maurizio Leotta ,&nbsp;Filippo Ricca ,&nbsp;Matteo Biagiola ,&nbsp;Paolo Tonella","doi":"10.1016/j.jss.2024.112304","DOIUrl":"10.1016/j.jss.2024.112304","url":null,"abstract":"<div><div>Web applications quality is commonly assessed by executing End-to-End (E2E) test scripts interacting with those systems as a human tester would. To avoid setting up the web application state for each test script, testers usually create test scripts that may depend on others previously executed. However, the presence of dependencies prevents parallelization, a fundamental technique for speedup the execution of large test suites.</div><div>In this paper, we present <span>Stile</span>, a tool for parallelizing the execution of E2E web test scripts that generates and executes a set of test schedules satisfying two important constraints: (1) every schedule respects existing test dependencies, and (2) all test scripts in the test suite are executed at least once. Moreover, <span>Stile</span> optimizes the execution by running only once the test scripts that are shared among the schedules.</div><div>We empirically evaluated <span>Stile</span> on eight E2E test suites by comparing the execution time of <span>Stile</span> both with the sequential execution and with the parallel execution based on Selenium Grid. Our results show that <span>Stile</span> can reduce the execution time up to 80% w.r.t. the sequential execution and up to 50% w.r.t. Grid. Moreover, <span>Stile</span> provides a reduction in the CPUs usage (i.e., overall CPU-time) up to 75%.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"222 ","pages":"Article 112304"},"PeriodicalIF":3.7,"publicationDate":"2024-12-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143104096","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}