Empirical Software Engineering最新文献

{"title":"MergeRepair: An Exploratory Study on Merging Task-Specific Adapters in Code LLMs for Automated Program Repair.","authors":"Meghdad Dehghan, Jie Jw Wu, Fatemeh H Fard, Ali Ouni","doi":"10.1007/s10664-026-10943-y","DOIUrl":"10.1007/s10664-026-10943-y","url":null,"abstract":"<p><p>Large Language Models (LLMs) have shown high capabilities in several software development-related tasks such as program repair, documentation, code refactoring, debugging, and testing. However, training these models requires massive amount of data and significant computational resources. Adapters are specialized, small modules designed for parameter efficient fine-tuning of LLMs for specific tasks, domains, or applications without requiring extensive retraining of the entire model. These adapters offer a more efficient way to customize LLMs for particular needs, leveraging the pre-existing capabilities of the large model. Model (and adapter) merging have emerged as a technique to develop one model capable of multiple tasks, with minimal or no training required. Although model and adapter merging has shown promising performance in domains such as natural language processing and computer vision, its applicability to software engineering tasks remains underexplored. In this paper, we investigate the effectiveness of merged adapters within the context of software engineering, with a particular focus on the Automated Program Repair (APR) task, through our approach, <b>MergeRepair</b>. In particular, we merge multiple task-specific adapters using three different merging methods, including weight-averaging, TIES-Merging, and DARE-TIES, and evaluate the performance of the merged adapter on the APR task. We introduce a <b>continual merging</b> approach, a novel method in which we sequentially merge the task-specific adapters where the order and weight of the merged adapters play a significant role. We further compare the performance of our approach with a baseline method consisting of equal-weight merging applied on parameters of different adapters, where all adapters are of equal importance. Through continual merging, we explore the capability of merged adapters and the effect of task order, as it occurs in real-world software projects. To evaluate our approach, we consider two LLMs, StarCoder2 and Granite models on an APR benchmark, HumanEvalFix, with various combinations of the merged adapters trained on a subset of the CommitPackFT dataset that contains four tasks and APR in Python. The results show that merging task-specific adapters can enhance the performance of models on the APR task, <i>without additional training</i>. In particular, our approach achieves up to 2.38% performance improvement over APR task in terms of pass@1 score when merging APR with Improvement and Misc task-adapters on top of the StarCoder2 model. Performance rises to 4.01% for pass@10 score with the same model and merged adapters. This improvement could be significantly influenced by the base model and the specific task adapters merged, rather than only the number of adapters. The proposed continual merging approach can be effective if the order of merging is efficiently determined. Results show that performance improves when the most effective task-adapter is added last","PeriodicalId":11525,"journal":{"name":"Empirical Software Engineering","volume":"32 1","pages":"37"},"PeriodicalIF":3.4,"publicationDate":"2027-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13615117/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149059353","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"How students use generative AI for software testing: An observational study.","authors":"Baris Ardic, Quentin Le Dilavrec, Andy Zaidman","doi":"10.1007/s10664-026-10898-0","DOIUrl":"10.1007/s10664-026-10898-0","url":null,"abstract":"<p><p>The integration of generative AI tools like ChatGPT into software engineering workflows opens up new opportunities to boost productivity in tasks such as unit test engineering. However, these AI-assisted workflows can also significantly alter the developer's role, raising concerns about control, output quality, and learning, particularly for novice developers. This study investigates how novice software developers with foundational knowledge in software testing interact with generative AI for engineering unit tests. Our goal is to examine the strategies they use, how heavily they rely on generative AI, and the benefits and challenges they perceive when using generative AI-assisted approaches for test engineering. We conducted an observational study involving 12 undergraduate students who worked with generative AI for unit testing tasks, using ChatGPT running the GPT-3.5 model. We identified four interaction strategies, defined by whether the test idea or the test implementation originated from generative AI or from the participant. Additionally, we singled out prompting styles that focused on one-shot or iterative test generation, which often aligned with the broader interaction strategy. Students reported benefits including time-saving, reduced cognitive load, and support for test ideation, but also noted drawbacks such as diminished trust, test quality concerns, and lack of ownership. While strategy and prompting styles influenced workflow dynamics, they did not significantly affect test effectiveness or test code quality as measured by mutation score or test smells.</p>","PeriodicalId":11525,"journal":{"name":"Empirical Software Engineering","volume":"31 6","pages":"167"},"PeriodicalIF":3.4,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13283165/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148301360","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Is common sense all you need? Using expert defined rules to identify vulnerability patches instead of machine learning.","authors":"Aurora Papotti, Serena Elisa Ponta, Antonino Sabetta, Fabio Massacci","doi":"10.1007/s10664-026-10860-0","DOIUrl":"https://doi.org/10.1007/s10664-026-10860-0","url":null,"abstract":"<p><strong>Goal: </strong>Machine learning (ML) has been proposed to identify security fixing commits with mixed success. We evaluate a different alternative in which expert-defined common sense rules with power-law weights are used to identify security fixing commit.</p><p><strong>Experiments: </strong>We first evaluated how those rules perform against ML models trained on the same features on which rules are based on the ProjectKB real world dataset of Java security commits. We then ran a think aloud protocol with seven senior analysts to analyze whether the selected rules are consistent with usage. Lastly, we ran the experiment with Master students ([Formula: see text]) to check whether the last ranking or the actual mention of which rules were applicable is beneficial to users with less experience. We found that the common-sense rules defined by security experts have a similar performance to classical ML approaches.</p><p><strong>Findings: </strong>Using Shap values to explain earned features, we find that ML models have the same chance corrected accuracy of expert defined rules, and they learned essentially the same top features identified by the experts and only differs on minor features (either among themselves and between the expert features). We observed that the juniors performance are comparable to the experts' one (CI [0.62, 0.76]). When asked to junior analysts to identify the fixing commits (in the top 10 selected by the tool) showing them which features was responsible for the selection do not seem to help (when compared with a control group with no support). A conclusion of our study is that one might just use ML to first analyze the data and then distill common sense rules that are still effective to apply. More experiments are needed to make features useful for the final decision.</p>","PeriodicalId":11525,"journal":{"name":"Empirical Software Engineering","volume":"31 5","pages":"133"},"PeriodicalIF":3.6,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13152979/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147867206","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"SecMLOps: A comprehensive framework for integrating security throughout the machine learning operations lifecycle.","authors":"Xinrui Zhang, Pincan Zhao, Jason Jaskolka, Heng Li, Rongxing Lu","doi":"10.1007/s10664-025-10795-y","DOIUrl":"10.1007/s10664-025-10795-y","url":null,"abstract":"<p><p>Machine Learning (ML) has emerged as a pivotal technology in the operation of large and complex systems, driving advancements in fields such as autonomous vehicles, healthcare diagnostics, and financial fraud detection. Despite its benefits, the deployment of ML models brings significant security challenges, such as adversarial attacks, which can compromise the integrity and reliability of these systems. To address these challenges, this paper builds upon the concept of Secure Machine Learning Operations (SecMLOps), providing a comprehensive framework designed to integrate robust security measures throughout the entire ML operations (MLOps) lifecycle. SecMLOps builds on the principles of MLOps by embedding security considerations from the initial design phase through to deployment and continuous monitoring. This framework is particularly focused on safeguarding against sophisticated attacks that target various stages of the MLOps lifecycle, thereby enhancing the resilience and trustworthiness of ML applications. A detailed advanced pedestrian detection system (PDS) use case demonstrates the practical application of SecMLOps in securing critical MLOps. Through extensive empirical evaluations, we highlight the trade-offs between security measures and system performance, providing critical insights into optimizing security without unduly impacting operational efficiency. Our findings underscore the importance of a balanced approach, offering valuable guidance for practitioners on how to achieve an optimal balance between security and performance in ML deployments across various domains.</p>","PeriodicalId":11525,"journal":{"name":"Empirical Software Engineering","volume":"31 3","pages":"74"},"PeriodicalIF":3.4,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12894179/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"146200483","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Less is more: usefulness of data flow diagrams and large language models for security threat validation.","authors":"Winnie Bahati Mbaka, Katja Tuma","doi":"10.1007/s10664-026-10837-z","DOIUrl":"10.1007/s10664-026-10837-z","url":null,"abstract":"<p><p>The arrival of recent cybersecurity standards has raised the bar for security assessments in organizations, but existing techniques require a high manual effort. Threat analysis and risk assessment are used to identify security threats for new or refactored systems. Still, there is a lack of definition-of-done, so identified threats have to be validated which slows down the analysis. Existing literature has focused on the overall effectiveness of threat analysis, but no previous work has investigated what material must the analysts use to effectively validate the identified security threats. We conduct a controlled experiment with practitioners to investigate whether having some analysis material (either the system's graphical model or LLM-generated advice) is better than none, and whether having both the system's graphical model and LLM-generated advice is better than having only one of them. We run a pilot of the experiment with 41 MSc students, a think-aloud study with three practitioners, and the experiment survey with 68 recruited practitioners. Our main findings suggest that, in terms of additional material needed for threat validation, less is more. We also find that participants perceived the graphical model as equally useful compared to LLMs and that, despite LLMs not always providing conclusive advice, practitioners still perceived it as somewhat useful. The experimental material and data analysis scripts is publicly available in a replication package.</p>","PeriodicalId":11525,"journal":{"name":"Empirical Software Engineering","volume":"31 5","pages":"122"},"PeriodicalIF":3.4,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13099727/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147766143","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Tools and benchmarks evolve: what is their impact on parameter tuning in SBSE experiments?","authors":"Amid Golmohammadi, Man Zhang, Andrea Arcuri","doi":"10.1007/s10664-025-10733-y","DOIUrl":"10.1007/s10664-025-10733-y","url":null,"abstract":"<p><p>In this article, we explore the impact of tool development and its evolution in Search-Based Software Engineering (SBSE) research. As a research tool evolves throughout the years, experiments with novel techniques might require reevaluation of previous studies, especially regarding parameter tuning. These reevaluations also give the opportunity to address the threats to external validity of these previous studies by employing a larger selection of artifacts. To conduct the replicated experiments in this study, the search-based fuzzer EvoMaster is chosen. This SBSE tool has been developed and extended throughout several years (since 2016) and tens of scientific studies. Among the chosen tool's parameters, 6 were carefully selected based on 5 previous studies that we replicate in this article with the latest version of EvoMaster. The replication is applied across an expanded set of artifacts compared to the original replicated studies. Our objective is to validate the robustness and validity of previous findings and to determine the need for parameter tuning in response to the tool's continuous development. Beyond replication, we explored parameter tuning by testing 729 different configurations to find a more performant parameter set, which is later validated through additional rounds of experiments. Additionally, we analyzed the impact of individual parameters on test generation performance using machine learning models, providing insights into their relative effects. Our findings indicate that, although most parameters maintain their efficacy, 2 of them require adjustment. Furthermore, the investigation into the effects of combining different parameter values reveals that carefully optimized configurations can outperform default settings. These findings highlight the importance of regularly reevaluating parameter settings to enhance tool performance in SBSE research.</p>","PeriodicalId":11525,"journal":{"name":"Empirical Software Engineering","volume":"31 1","pages":"8"},"PeriodicalIF":3.4,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12583389/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145451308","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Developers and generative AI: A study of self-admitted usage in open source projects.","authors":"Rosalia Tufano, Federica Pepe, Fiorella Zampetti, Antonio Mastropaolo, Ozren Dabić, Massimiliano Di Penta, Gabriele Bavota","doi":"10.1007/s10664-026-10848-w","DOIUrl":"https://doi.org/10.1007/s10664-026-10848-w","url":null,"abstract":"<p><p>The availability of generative Artificial Intelligence (AI) tools such as ChatGPT or GitHub Copilot is reshaping the way in which software is developed, evolved, and maintained. Oftentimes, developers leave traces of such an usage in software artifacts. This allows not only to understand how AI is used in software development, but also to let others be aware how such software artifacts were created, e.g., for licensing or trustworthiness purposes. This paper-building upon our preliminary work presented at MSR 2024-aims at qualitatively investigating on the self-admitted use of two very popular generative AI tools-ChatGPT and GitHub Copilot-in software development. To this aim, we mined GitHub for such traces, by looking at commits, issues and pull requests (PRs). Then, through a manual coding, we create a taxonomy of 64 different ChatGPT and GitHub Copilot usage tasks, grouped into 7 categories. By repeating our previous analysis two years after and by extending it to GitHub Copilot, we show how the usage avenues have been expanded, the extent to which developers perceived such a generative AI usage useful, and whether some concerns occurring more than one year ago are no longer present. The taxonomy of tasks we derived from such a qualitative study provided (i) developers with valuable insights into how generative AI can be integrated into their workflows, and (ii) researchers with a clear overview of tasks that developers perceive as well-suited for automation.</p>","PeriodicalId":11525,"journal":{"name":"Empirical Software Engineering","volume":"31 4","pages":"108"},"PeriodicalIF":3.4,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13658386/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149179070","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Simplifying software compliance: AI technologies in drafting technical documentation for the AI Act.","authors":"Francesco Sovrano, Emmie Hine, Stefano Anzolut, Alberto Bacchelli","doi":"10.1007/s10664-025-10645-x","DOIUrl":"10.1007/s10664-025-10645-x","url":null,"abstract":"<p><p>The European AI Act has introduced specific technical documentation requirements for AI systems. Compliance with them is challenging due to the need for advanced knowledge of both legal and technical aspects, which is rare among software developers and legal professionals. Consequently, small and medium-sized enterprises may face high costs in meeting these requirements. In this study, we explore how contemporary AI technologies, including ChatGPT and an existing compliance tool (DoXpert), can aid software developers in creating technical documentation that complies with the AI Act. We specifically demonstrate how these AI tools can identify gaps in existing documentation according to the provisions of the AI Act. Using open-source high-risk AI systems as case studies, we collaborated with legal experts to evaluate how closely tool-generated assessments align with expert opinions. Findings show partial alignment, important issues with ChatGPT (3.5 and 4), and a moderate (and statistically significant) correlation between DoXpert and expert judgments, according to the Rank Biserial Correlation analysis. Nonetheless, these findings underscore the potential of AI to combine with human analysis and alleviate the compliance burden, supporting the broader goal of fostering responsible and transparent AI development under emerging regulatory frameworks.</p>","PeriodicalId":11525,"journal":{"name":"Empirical Software Engineering","volume":"30 3","pages":"91"},"PeriodicalIF":3.5,"publicationDate":"2025-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11965209/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143794942","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"On the effects of program slicing for vulnerability detection during code inspection.","authors":"Aurora Papotti, Katja Tuma, Fabio Massacci","doi":"10.1007/s10664-025-10636-y","DOIUrl":"10.1007/s10664-025-10636-y","url":null,"abstract":"<p><p>Slicing is a fault localization technique that has been proposed to support debugging and program comprehension. Yet, its empirical effectiveness during code inspection by humans has received limited attention. The goal of our study is two-fold. First, we aim to define what it means for a code reviewer to identify the vulnerable lines correctly. Second, we investigate whether reducing the number of to-be-inspected lines by method-level slicing supports code reviewers in detecting security vulnerabilities. We propose a novel approach based on the notion of a <math><mi>δ</mi></math> -neighborhood (intuitively based on the idea of the context size of the command git  diff) to define correctly identified lines. Then, we conducted a multi-year controlled experiment (2017-2023) in which MSc students attending security courses ( <math><mrow><mi>n</mi> <mo>=</mo> <mn>236</mn></mrow> </math> ) were tasked with identifying vulnerable lines in original or sliced Java files from Apache Tomcat. We provide perfect seed lines for a slicing algorithm to control for confounding factors. Each treatment differs in the pair (Vulnerability, Original/Sliced) with a balanced design with vulnerabilities from the OWASP Top 10 2017: A1 (Injection), A5 (Broken Access Control), A6 (Security Misconfiguration), and A7 (Cross-Site Scripting). To generate smaller slices for human consumption, we used a variant of intra-procedural thin slicing. We report the results for <math><mrow><mi>δ</mi> <mo>=</mo> <mn>0</mn></mrow> </math> which corresponds to exactly matching the vulnerable ground truth lines, and <math><mrow><mi>δ</mi> <mo>=</mo> <mn>3</mn></mrow> </math> which represents the scenario of identifying the vulnerable area. For both cases, we found that slicing helps in 'finding something' (the participant has found at least some vulnerable lines) as opposed to 'finding nothing'. For the case of <math><mrow><mi>δ</mi> <mo>=</mo> <mn>0</mn></mrow> </math> analyzing a slice and analyzing the original file are statistically equivalent from the perspective of lines found by those who found something. With <math><mrow><mi>δ</mi> <mo>=</mo> <mn>3</mn></mrow> </math> slicing helps to find more vulnerabilities compared to analyzing an original file, as we would normally expect. Given the type of population, additional experiments are necessary to be generalized to experienced developers.</p>","PeriodicalId":11525,"journal":{"name":"Empirical Software Engineering","volume":"30 3","pages":"93"},"PeriodicalIF":3.5,"publicationDate":"2025-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11972194/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143802692","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"An empirical study of fault localisation techniques for deep neural networks.","authors":"Nargiz Humbatova, Jinhan Kim, Gunel Jahangirova, Shin Yoo, Paolo Tonella","doi":"10.1007/s10664-025-10657-7","DOIUrl":"10.1007/s10664-025-10657-7","url":null,"abstract":"<p><p>With the increased popularity of Deep Neural Networks (DNNs), increases also the need for tools to assist developers in the DNN implementation, testing and debugging process. Several approaches have been proposed that automatically analyse and localise potential faults in DNNs under test. In this work, we evaluate and compare existing state-of-the-art fault localisation techniques, which operate based on both dynamic and static analysis of the DNN. The evaluation is performed on a benchmark consisting of both real faults obtained from bug reporting platforms and faulty models produced by a mutation tool. Our findings indicate that the usage of a single, specific ground truth (e.g. the human-defined one) for the evaluation of DNN fault localisation tools results in pretty low performance (maximum average recall of 0.33 and precision of 0.21). However, such figures increase when considering alternative, equivalent patches that exist for a given faulty DNN. The results indicate that DeepFD is the most effective tool, achieving an average recall of 0.55 and a precision of 0.37 on our benchmark.</p>","PeriodicalId":11525,"journal":{"name":"Empirical Software Engineering","volume":"30 5","pages":"124"},"PeriodicalIF":3.6,"publicationDate":"2025-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12152046/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144283001","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}