Journal of Systems and Software最新文献

{"title":"Is our chatbot telling lies? Assessing correctness of an LLM-based Dutch support chatbot","authors":"Herman Lassche ,&nbsp;Michiel Overeem ,&nbsp;Ayushi Rastogi","doi":"10.1016/j.jss.2025.112699","DOIUrl":"10.1016/j.jss.2025.112699","url":null,"abstract":"<div><div>Companies support their customers using live chats and chatbots to gain their loyalty. AFAS is a Dutch company aiming to leverage the opportunity large language models (LLMs) offer to answer customer queries with minimal to no input from its customer support team. Adding to its complexity, it is unclear what makes a response correct, and that too in Dutch. Further, with minimal data available for training, the challenge is to identify whether an answer generated by a large language model is correct and do it on the fly.</div><div>This study is the first to define the correctness of a response based on how the support team at AFAS makes decisions. It leverages literature on natural language generation and automated answer grading systems to automate the decision-making of the customer support team. We investigated questions requiring a binary response (e.g., Would it be possible to adjust tax rates manually?) or instructions (e.g., How would I adjust tax rate manually?) to test how close our automated approach reaches support rating. Our approach can identify wrong messages in 55 % of the cases. This work demonstrates the potential for automatically assessing when our chatbot may provide incorrect or misleading answers. Specifically, we contribute (1) a definition and metrics for assessing correctness, and (2) suggestions to improve correctness with respect to regional language and question type.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"234 ","pages":"Article 112699"},"PeriodicalIF":4.1,"publicationDate":"2026-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145685641","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Can large language models identify and refactor code clones? An empirical study","authors":"Xing Qian,&nbsp;Eman Abdullah AlOmar","doi":"10.1016/j.jss.2025.112717","DOIUrl":"10.1016/j.jss.2025.112717","url":null,"abstract":"<div><div>Large Language Models (LLMs), such as GPT-4, have been widely used in various software engineering tasks, including code generation, code clone detection, refactoring and code review. Recent studies have compared the capability of different LLMs on code clone detection, especially the more complex Type-4 clones. However, despite all those studies, little is known about how LLMs perform on clone classification and refactoring compared to the state-of-the-art tools. In this paper, we explore how is GPT-4o’s performance when we consider the results of traditional method JDeodorant as ground truth. We categorized clones into different scenarios, such as whole method clones within the same file, whole method clones in different files, code fragment clones within the same file and code fragment clones in different files. Then, we compared the refactoring capability of LLMs, on different scenarios with JDeodorant based on compilation rate, correctness and the use of lambda expressions. We further expanded our evaluation to include a diverse set of LLMs, including Gemini 2.5 Pro, GPT-3.5, GPT-4o, GPT-5, DeepSeek-R1, Claude Sonnet 3.7, and Claude Opus 4.1, along with traditional tools such as JDeodorant and AntiCopyPaster. These models and tools were applied to 20 cases from the JHotDraw project, covering both Type-1 and Type-2 code clones. Our results show that GPT-4o achieved the perfect performance to identify Type-1 clones, but the accuracy dropped to around 51.2 % for Type-2 clones and around 85.6 % for Type-3 clones. Our findings reveal that while GPT-4o performs well on Type-1 clone classification, it struggles to identify Type-2 and Type-3 clones. For refactoring, GPT-4o achieved good results (84.2 % correctness rate) when refactoring Type-1 clones. However, when clones are Type-2 or Type-3, the correctness of GPT dropped significantly, where it only has 54.6 % for Type-2 and 33.1 % for Type-3, indicating that GPT-4o struggles to refactor complex cases. When it comes to our case study, our results reveal a clear contrast between traditional tools and LLMs. JDeodorant and AntiCopyPaster consistently produced correct refactorings, with all clones detected, compiled, and tests passed. Among LLMs, GPT-4o and GPT-5 stood out, achieving 14 and 15 correct refactorings respectively, while models such as Gemini 2.5 Pro, Claude Sonnet 3.7, and Claude Opus 4.1 showed partial success but often generated incomplete or fragmented changes. DeepSeek-R1 produced mixed results, sometimes replacing clones with calls but also introducing fragmentation of logic. In contrast, GPT-3.5 failed to detect all the clones. We anticipate our work could contribute to a boarder understanding of the capabilities of LLMs on code clone detection and refactoring.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"234 ","pages":"Article 112717"},"PeriodicalIF":4.1,"publicationDate":"2026-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145840747","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A consistency management framework for digital twin models","authors":"Hossain Muhammad Muctadir ,&nbsp;Eduard Kamburjan ,&nbsp;Loek Cleophas ,&nbsp;Mark van den Brand","doi":"10.1016/j.jss.2025.112750","DOIUrl":"10.1016/j.jss.2025.112750","url":null,"abstract":"<div><div>Digital twins (DTs) encapsulate the concept of a real-world entity (RE) and corresponding bidirectionally connected virtual one (VE) mimicking certain aspects of the former in order to facilitate various use-cases such as predictive maintenance. DTs typically encompass various models that are often developed by experts from different domains using diverse tools. To maintain consistency among these models and ensure the continued functioning of the system, effective identification of any consistency issues and addressing them whenever necessary is imperative. In this paper, we investigate the concept of consistency management and propose a consistency management framework that addresses various characteristics of DT models. Subsequently, we present three working examples that implement the proposed framework with graph-based techniques. Taking the working examples into account, we demonstrate and argue that our consistency management framework can provide crucial assistance in the consistency management of DT models.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"234 ","pages":"Article 112750"},"PeriodicalIF":4.1,"publicationDate":"2026-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145884145","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Applying generative artificial intelligence for vulnerability fixing in a proprietary software ecosystem","authors":"Luiz Alexandre Costa ,&nbsp;Awdren Fontão ,&nbsp;Rodrigo Pereira dos Santos ,&nbsp;Alexander Serebrenik","doi":"10.1016/j.jss.2025.112723","DOIUrl":"10.1016/j.jss.2025.112723","url":null,"abstract":"<div><div><strong>Context:</strong> Large organizations often operate within proprietary software ecosystems (PSECO), composed of interdependent software artifacts, multiple actors, and centralized governance. In such environments, addressing security vulnerabilities is particularly challenging due to strict quality standards, regulatory constraints, and the risk of cascading failures. These challenges can compromise delivery schedules, increase operational risk, and force teams to interrupt planned work to address urgent issues. <strong>Goal:</strong> In response to these challenges, recent advances in Generative Artificial Intelligence (GenAI), particularly large language models (LLM), have emerged as a promising avenue to support software engineering tasks such as code generation and automated repair. This study investigates how a GenAI-based approach can support vulnerability remediation in PSECO while preserving delivery cadence in Continuous Integration and Continuous Delivery (CI/CD) pipelines. <strong>Methods:</strong> We conducted a participative case study in a large global organization to design and evaluate PSECO-SafePatch, an approach composed of: (i) a structured remediation process tailored to enterprise constraints; and (ii) a web-based tool integrating Fortify static analysis with GenAI-based patch generation. The approach includes human-in-the-loop validation and aligns with DevOps practices. <strong>Results:</strong> PSECO-SafePatch reduced mean remediation time by 84 % with an 89 % patch success rate. It also reduced cognitive overload by guiding developers through structured validation, fostering trust without overreliance. <strong>Conclusion:</strong> The findings show that GenAI-supported remediation is feasible and effective in PSECO. Human-in-the-loop validation preserved critical thinking, addressing concerns about blind automation and knowledge erosion, and reinforcing the value of responsible automation at scale.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"234 ","pages":"Article 112723"},"PeriodicalIF":4.1,"publicationDate":"2026-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145790956","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A knowledge graph enabled recommendation system for implicitly associated items: Application to vertical e-commerce of parts","authors":"Xinjun Lai ,&nbsp;Guitao Huang ,&nbsp;Yirun Chen ,&nbsp;Dejun Wang ,&nbsp;Martin Lai ,&nbsp;Ming Cai","doi":"10.1016/j.jss.2025.112698","DOIUrl":"10.1016/j.jss.2025.112698","url":null,"abstract":"<div><div>Although recommendation algorithms have been proved efficient for the e-commerce platforms in the last twenty years, it is not a trivial task for the vertical platforms to adopt these methods. Compared to the general platforms, mostly, the vertical ones share the following characteristics: (1) data volume is small but data types are various, due to the online ecosystem and community; (2) items might be implicitly associated, such as parts; (3) operated by SME (small and medium-size enterprise), where the IT and AI resources are limited. Targeted at these features, we propose a knowledge-graph(KG)-based recommender for our studied Lego parts company. First, a KG is developed to mine the various implicit associations of users and items, where information on users’/designers’ online works, posts, interactions, buying behaviours etc., and part-set relations, are modelled in the heterogeneous KG. Second, a modified RippleNet algorithm is proposed, where users’ interests are modelled as ripples in the KG. In addition, information on important neighbouring nodes is embedded, to model the semi-social influence in the KG for a user. Third, the best timing to update the algorithm is studied by monitoring and predicting the topology of the KG, to achieve the best cost-performance in algorithm operation and maintenance. The recommender system is implemented in the studied company, where the offline and online evaluations suggest that our method is practical, efficient and SME-friendly.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"233 ","pages":"Article 112698"},"PeriodicalIF":4.1,"publicationDate":"2026-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145618386","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"GraMuS: Boosting statement-level fault localization via graph representation and multimodal information","authors":"Ruishi Huang ,&nbsp;Binbin Yang ,&nbsp;Shumei Wu ,&nbsp;Zheng Li ,&nbsp;Doyle Paul ,&nbsp;Xiao-Yi Zhang ,&nbsp;Xiang Chen ,&nbsp;Yong Liu","doi":"10.1016/j.jss.2025.112700","DOIUrl":"10.1016/j.jss.2025.112700","url":null,"abstract":"<div><div>Fault Localization (FL) aims to reduce the cost of manual debugging by highlighting the statements which are more likely responsible for observed failures. However, existing techniques have limited effectiveness in practice due to inflexible suspiciousness evaluations and oversimplified representation of execution information. In this paper, we propose GraMuS, a novel <em><strong>Gra</strong></em>ph representation learning and <em><strong>Mu</strong></em>ltimodal information based technique for <em><strong>S</strong></em>tatement-level FL. GraMuS comprises two key components: a fine-grained fault diagnosis graph and a multi-level collaborative suspiciousness evaluation. The former integrally records enriched multimodal information from various levels of granularity (including methods, statements, and mutants) by a graph structure. The latter utilizes the interactions between FL tasks at various levels of granularity to extract existing/latent useful features from multimodal information for improving FL precision. Empirical studies on the widely used Defects4J(V2.0.0) dataset show that GraMuS can outperform state-of-the-art baselines in both single-fault programs and multiple-fault programs, including one large language models, four learning-based FL techniques, three variable-based FL techniques, 36 spectrum-based FL techniques, and 36 mutation-based FL techniques. In particular, GraMuS can localize 26/29/31 more faulty statements than the state-of-the-art baseline ChatGPT-4/DepGraph/VarDT, in terms of <span><math><mrow><mi>T</mi><mi>O</mi><mi>P</mi><mspace></mspace><mo>−</mo><mspace></mspace><mn>1</mn></mrow></math></span> metric. Further investigation shows that the method-level FL task can help GraMuS localize 27 more faulty statements, resulting in a 50.94 % improvement. Finally, we further evaluate GraMuS in 374 Python programs from ConDefects, and find that GraMuS consistently outperforms state-of-the-art FL techniques, showing its generality.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"233 ","pages":"Article 112700"},"PeriodicalIF":4.1,"publicationDate":"2026-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145618389","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"User stories as boundary objects in agile requirements engineering: A theoretical literature review","authors":"Tor Sporsem ,&nbsp;Torgeir Dingsøyr ,&nbsp;Klaas-Jan Stol","doi":"10.1016/j.jss.2025.112693","DOIUrl":"10.1016/j.jss.2025.112693","url":null,"abstract":"<div><div>User stories have become the predominant method for managing requirements in software development, used by approximately half of all software developers. Despite this widespread adoption, there is limited theoretical understanding of how user stories are used in practice. Through a theoretical literature review of 14 industry studies, we develop five theoretical propositions: 1) user stories facilitate shared understanding between developers and users; 2) small user stories help developers cope with change; 3) clarifying the ‘why’ in user stories reinforces focus on user needs but adds complexity to the development process; 4) conversations triggered by user stories can hamper the sense of productivity; and 5) user stories as recorded in writing degrade over time. Using boundary object theory as an analytical lens, we explain how user stories facilitate knowledge transfer across syntactic, semantic, and pragmatic boundaries between developers and users. This theoretical lens offers new insights into why some user stories succeed while others fail to bridge boundaries between users and developers. The review highlights the sharp contrast between the widespread use of user stories among practitioners and the limited academic research on their practical application. We end with identifying opportunities for future research, particularly on how user stories can be used in the era of generative AI.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"233 ","pages":"Article 112693"},"PeriodicalIF":4.1,"publicationDate":"2026-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145618391","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Evaluating correctness, performance and energy footprint of semantic reasoners in mobile edge computing","authors":"Ivano Bilenchi,&nbsp;Davide Loconte,&nbsp;Floriano Scioscia,&nbsp;Michele Ruta","doi":"10.1016/j.jss.2025.112696","DOIUrl":"10.1016/j.jss.2025.112696","url":null,"abstract":"<div><div>The integration of Semantic Web technologies into Mobile Edge Computing (MEC) platforms is enhancing the capabilities of real-time, context-aware applications across diverse domains. MEC brings processing closer to the network edge, reducing latency and allowing for the improvement of data privacy, while Semantic Web technologies provide machine-interpretable knowledge representation and reasoning capabilities. Despite their potential, deploying semantic reasoners on edge devices is challenging due to their resource-intensive nature, which requires significant memory availability, computational power, and energy. Furthermore, correctness, performance and energy consumption are simultaneously important, as MEC semantics-based applications often call for real-time queries for autonomous agent decision or user-oriented decision support. This paper presents an extensive experimental evaluation of Web Ontology Language (OWL) reasoners deployed in MEC environments, assessing correctness, processing time, memory usage, and energy consumption across both a reference tablet and a single-board computer. For energy measurement, both software profiling and hardware monitoring have been exploited and compared. The study is supported by a modular, cross-platform benchmarking framework that automates data collection and ensures reproducibility. The findings highlight the trade-offs between reasoning capabilities and resource consumption, offering valuable insights for refining testing methodologies as well as optimizing semantic reasoners in MEC settings.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"233 ","pages":"Article 112696"},"PeriodicalIF":4.1,"publicationDate":"2026-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145572020","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Decision making for automotive-platform engineering: A mixed-methods study on practitioners’ requirements and criteria","authors":"Philipp Zellmer ,&nbsp;Jacob Krüger ,&nbsp;Thomas Leich","doi":"10.1016/j.jss.2025.112670","DOIUrl":"10.1016/j.jss.2025.112670","url":null,"abstract":"<div><div>With modern vehicles becoming increasingly software driven, long-standing automotive manufacturers face challenges in integrating software into their historically hardware-centered platforms. Even though hardware and software platforms leverage the same ideas, manufacturers must still unify hardware, software, and electrics/electronics components; as well as their intricate interconnections and variability. To address this challenge, automotive manufacturers are adopting product-structuring and variant-management concepts from different areas to develop a holistic platform perspective. Here, particularly decision making has manifested as a key problem to inform the involved stakeholders and assess different options. Specifically, simply adding software to a static hardware platform is insufficient to address the current digital transformation in the automotive industry: the new platform dimensions must be understood, changes balanced against each other, and necessary rework defined to achieve holistic platform management. In this article, we contribute to tackling this problem by presenting a mixed-method study through which we derived a set of requirements to support decision-making for today’s automotive electrics/electronics platforms. We used requirements and insights from our previous work, which we refined and validated via expert workshops and a questionnaire with 76 experts from the automotive industry. Besides confirming and validating our previous work, we contribute a new consolidated overview of requirements and criteria together with a prioritization by practitioners. The findings of our work offer valuable guidance for automotive practitioners, particularly for adopting product-structuring concepts like electrics/electronics platforms and making decisions in the process. Our contributions further substantiate existing evidence and can guide research on developing decision-making techniques for complex electrics/electronics platforms.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"233 ","pages":"Article 112670"},"PeriodicalIF":4.1,"publicationDate":"2026-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145555034","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Dear Researchers: Think about the future, for sure, but please don't forget about the present – the Odense Manifesto for academia–industry collaboration at ICSA 2025","authors":"Jens Bæk Jørgensen ,&nbsp;Morten Jokumsen ,&nbsp;Romina Spalazzese","doi":"10.1016/j.jss.2025.112681","DOIUrl":"10.1016/j.jss.2025.112681","url":null,"abstract":"","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"233 ","pages":"Article 112681"},"PeriodicalIF":4.1,"publicationDate":"2026-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145685132","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}