Journal of Systems and Software最新文献

{"title":"LeCov: Multi-level testing criteria for large language models","authors":"Xuan Xie ,&nbsp;Jiayang Song ,&nbsp;Yuheng Huang ,&nbsp;Da Song ,&nbsp;Felix Juefei-Xu ,&nbsp;Lei Ma","doi":"10.1016/j.jss.2025.112763","DOIUrl":"10.1016/j.jss.2025.112763","url":null,"abstract":"<div><div>Large Language Models (LLMs) are widely used in many different domains, but because of their limited interpretability, there are questions about how trustworthy they are in various perspectives, e.g., truthfulness and toxicity. Recent research has started developing testing methods for LLMs, aiming to uncover untrustworthy issues, i.e., defects, before deployment. However, systematic and formalized testing criteria are lacking, which hinders a comprehensive assessment of the extent and adequacy of testing exploration. To mitigate this threat, we propose a set of multi-level testing criteria, LeCov, for LLMs. The criteria consider three crucial LLM internal components, i.e., the attention mechanism, feed-forward neurons, and uncertainty, and contain nine types of testing criteria in total. We apply the criteria in two scenarios: test prioritization and coverage-guided testing. The experiment evaluation, on five models and four datasets, demonstrates the usefulness and effectiveness of LeCov.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112763"},"PeriodicalIF":4.1,"publicationDate":"2026-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145928356","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Ethics label for digital systems to promote transparency and user awareness","authors":"Marco Autili ,&nbsp;Riccardo Corsi ,&nbsp;Martina De Sanctis ,&nbsp;Paola Inverardi ,&nbsp;Patrizio Pelliccione","doi":"10.1016/j.jss.2025.112752","DOIUrl":"10.1016/j.jss.2025.112752","url":null,"abstract":"<div><div>Modern digital systems pose risks to humans, society, and the environment. There is a flourishing of guidelines, recommendations, laws, and regulations, but also of standards that try to regulate and alleviate the lack of good practices for the governance, management, and quality of AI systems. However, having just a mark certifying that the system passed some checks is not enough and is fragile to the ethics washing problem. The objective of this work is to go beyond compliance to standards toward an ethics label that enables users to understand the impact of systems on human, societal, and environmental values, both during system development and usage. To build the ethics label, we analyze guidelines, recommendations, laws and regulations, and quality standards. We validate the proposed ethics label through (i) a proof-of-concept application to the social assistive robotics domain, and (ii) interviews with experts from both academia and industry. We contribute an ethics label for modern digital systems that promote transparency and user awareness, and enable users to select systems that meet their subjective ethical preferences. We discuss the problem of ethics washing and propose an ethics label as a means to foster transparency and raise user awareness.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112752"},"PeriodicalIF":4.1,"publicationDate":"2026-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"146038341","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"ParserHunter: Identify parsing functions in binary code","authors":"Marco Scapin,&nbsp;Fabio Pinelli,&nbsp;Letterio Galletta","doi":"10.1016/j.jss.2026.112783","DOIUrl":"10.1016/j.jss.2026.112783","url":null,"abstract":"<div><div>Parsing and validation functions are crucial because they process untrusted data, e.g., user inputs. Due to their complexity, these functions are highly susceptible to bugs, making them a primary target for security audits. However, identifying such functions within a binary is time-intensive and challenging, given the numerous functions typically present and the lack of source code or supporting documentation. This paper presents an AI-based methodology for identifying functions with parser-like behavior and complex processing logic within a binary. Our methodology analyzes each binary by identifying its functions, extracting their Control Flow Graphs (CFGs), and enriching them with features derived from an embedding model that captures both structural and semantic aspects of their behavior. These annotated CFGs are the input to a Graph Neural Network trained to identify parsing functions. We implement this methodology in the tool ParserHunter, which allows users to train the model on labeled data, query the model with unseen binaries, and accommodate a symbolic execution phase on the processed binary through a user interface. Our experiments on ten real-world projects from GitHub show that our tool effectively identifies parsers in binaries.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112783"},"PeriodicalIF":4.1,"publicationDate":"2026-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145978634","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"RLV: LLM-based vulnerability detection by retrieving and refining contextual information","authors":"Fangcheng Qiu ,&nbsp;Zhongxin Liu ,&nbsp;Bingde Hu ,&nbsp;Zhengong Cai ,&nbsp;Lingfeng Bao ,&nbsp;Xinyu Wang","doi":"10.1016/j.jss.2025.112756","DOIUrl":"10.1016/j.jss.2025.112756","url":null,"abstract":"<div><div>Vulnerability detection plays a critical role in ensuring software quality during the processes of software development and maintenance. Automated vulnerability detection methods have been proposed to reduce the consumption of human and material resources. From traditional machine learning-based approaches to deep learning-based approaches, vulnerability detection techniques have continuously evolved and improved. Recently, Large Language Models (LLMs) have been increasingly applied to vulnerability detection. However, deep learning-based approaches and LLM-based approaches suffer from two main problems: (1) They suffer from poor generalization capabilities, which limit their performance in real-world scenarios. (2) They lack accurate contextual information of the target function, which hinders their ability to correctly understand the target function. To tackle these problems, in this paper, we propose a novel vulnerability detection approach, named <span>RLV</span> (<u><strong>R</strong></u>etrieving&amp;Refining Contextual Information for <u><strong>L</strong></u>LM-based <u><strong>V</strong></u>ulnerability Detection), an LLM-based approach that enhances vulnerability detection by integrating project-level contextual information into the analysis process. RLV emulates how programmers reason about code, enabling the LLM to retrieve and refine relevant semantic context from the project repository to better understand the target function. Besides, RLV guides the LLM via effective prompts, avoiding task-specific training and enhancing its practicality in real-world scenarios. We conduct experiments on two vulnerability datasets with a total of 30,436 vulnerable functions and 306,269 non-vulnerable functions. The experimental results demonstrate that our approach achieves state-of-the-art performance. Moreover, our approach achieves a 26.83% improvement in terms of <em>F</em>₁-score over state-of-the-art baselines when tested on unseen projects.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112756"},"PeriodicalIF":4.1,"publicationDate":"2026-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145978693","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A white-box prompt injection attack on embodied AI agents driven by large language models the","authors":"Tongcheng Geng ,&nbsp;Yubin Qu ,&nbsp;W. Eric Wong","doi":"10.1016/j.jss.2026.112782","DOIUrl":"10.1016/j.jss.2026.112782","url":null,"abstract":"<div><div>With the widespread deployment of embodied AI agents in safety-critical scenarios, LLM-based decision-making systems face unprecedented risks. Existing prompt injection attacks, designed for general conversational systems, lack semantic contextual adaptability for embodied agents and fail to address scenario-specific semantics and safety constraints. This paper proposes <strong>SAPIA</strong> (<strong>S</strong>cenario-<strong>A</strong>daptive white-box <strong>P</strong>rompt <strong>I</strong>njection <strong>A</strong>ttack), integrating an adaptive context prompt generation module with an enhanced GCG algorithm to dynamically produce scenario-targeted adversarial suffixes. We build a multi-scenario dataset of 40 dangerous instructions across four application domains–autonomous driving, robotic manipulation, drone control, and industrial control–establishing a standardized benchmark for embodied AI safety. Large-scale white-box experiments on three mainstream open-source LLMs show SAPIA substantially outperforms traditional GCG and improved I-GCG, with notably high effectiveness on extremely high-risk instructions. Transferability analysis reveals distinctive properties in embodied settings: cross-architecture transfer is extremely limited, while high cross-version transferability exists within model series, contrasting with cross-model transfer observed in conventional adversarial research. Ablation studies confirm both the adaptive context module and enhanced GCG are critical and synergistic for optimal attack performance. Robustness analyses indicate SAPIA strongly resists mainstream defenses, effectively evading input perturbation, structured self-examination, and safety prefix prompting. This work exposes serious security vulnerabilities in current embodied AI agents and underscores the urgency of scenario-based protection mechanisms for safety-critical deployments.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112782"},"PeriodicalIF":4.1,"publicationDate":"2026-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145978632","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"LogMeta: A few-shot model-agnostic meta-learning framework for robust and adaptive log anomaly detection","authors":"Yicheng Sun,&nbsp;Jacky Wai Keung,&nbsp;Hi Kuen Yu,&nbsp;Wenqiang Luo","doi":"10.1016/j.jss.2026.112781","DOIUrl":"10.1016/j.jss.2026.112781","url":null,"abstract":"<div><div><strong>Context:</strong> Log anomaly detection is critical for maintaining the security, stability, and operational efficiency of modern software systems, especially as they generate vast and diverse log data. However, existing deep learning models struggle with the challenges of heterogeneous log formats across systems and the scarcity of labeled anomaly logs, limiting their real-world deployment and generalization capabilities.</div><div><strong>Objective:</strong> To address these challenges, we propose LogMeta, a novel semi-supervised framework designed for adaptive and efficient log anomaly detection in diverse and low-resource environments.</div><div><strong>Method:</strong> LogMeta integrates Model-Agnostic Meta-Learning (MAML) with a hybrid language model to address key challenges. MAML enables LogMeta to rapidly adapt to unseen log systems using few-shot samples, while the hybrid model combines RoBERTa for extracting semantic representations with Bi-LSTM and attention mechanisms to capture sequential dependencies and critical features within log sequences. This design reduces reliance on large-scale labeled datasets and enhances adaptability in heterogeneous environments.</div><div><strong>Results:</strong> Experimental evaluations on multiple benchmark datasets demonstrate that LogMeta consistently outperforms state-of-the-art supervised and unsupervised methods, achieving up to a 28.3% improvement in F1-scores under low-resource scenarios compared to other models. Furthermore, LogMeta exhibits exceptional domain transfer capabilities, maintaining robust performance across diverse log datasets with minimal fine-tuning. In terms of efficiency, LogMeta achieves competitive training and inference times, making it suitable for real-time anomaly detection in large-scale systems.</div><div><strong>Conclusion:</strong> LogMeta provides a scalable and practical solution for real-world log anomaly detection, overcoming challenges related to data heterogeneity and label scarcity. Its strong generalization capabilities, minimal supervision requirements, and adaptability to new log systems make it a promising tool for enhancing software system reliability and security.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112781"},"PeriodicalIF":4.1,"publicationDate":"2026-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145978692","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A systematic literature review of software engineering research on Jupyter notebook","authors":"Md Saeed Siddik ,&nbsp;Hao Li ,&nbsp;Cor-Paul Bezemer","doi":"10.1016/j.jss.2025.112758","DOIUrl":"10.1016/j.jss.2025.112758","url":null,"abstract":"<div><div><em>Context</em>: Jupyter Notebook has emerged as a versatile tool that transforms how researchers, developers, and data scientists conduct and communicate their work. As the adoption of Jupyter notebooks continues to rise, so does the interest from the software engineering research community in improving the software engineering practices for Jupyter notebooks.</div><div><em>Objective</em>: The purpose of this study is to analyze trends, gaps, and methodologies used in software engineering research on Jupyter notebooks.</div><div><em>Method</em>: We selected 199 relevant publications up to September 2025, following established systematic literature review guidelines. We explored publication trends, categorized them based on software engineering topics, and reported findings based on those topics.</div><div><em>Results</em>: The most popular venues for publishing software engineering research on Jupyter notebooks are related to human-computer interaction instead of traditional software engineering venues. Researchers have addressed a wide range of software engineering topics on notebooks, such as code reuse, readability, and execution environment. Although reusability is one of the research topics for Jupyter notebooks, only 82 of the 199 studies can be reused based on their provided URLs. Additionally, most replication packages are not hosted on permanent repositories for long-term availability and adherence to open science principles.</div><div><em>Conclusion</em>: Solutions specific to notebooks for software engineering issues, including testing, refactoring, and documentation, are underexplored. Future research opportunities exist in automatic testing frameworks, refactoring clones between notebooks, and generating group documentation for coherent code cells.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112758"},"PeriodicalIF":4.1,"publicationDate":"2026-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145852497","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"On software testing reference ontologies","authors":"Maryam Havakeshian,&nbsp;Yvan Labiche","doi":"10.1016/j.jss.2025.112759","DOIUrl":"10.1016/j.jss.2025.112759","url":null,"abstract":"<div><div>Software testing manipulates various artifacts, such as application and test code, requirements, test objectives and results, that are important to stakeholders such as developers, testers, and the QA team. To reason and make decisions from these varied artifacts, information needs to be recorded in a structured, computer-readable format. This highlights the need for effective knowledge management, for which ontologies are an ideal solution.</div><div>We aim to answer two key questions for software testing practitioners who wish to use ontology-based knowledge management: what criteria can one rely on to decide which testing ontology to use, and how do existing ontologies fare when using these criteria?</div><div>We coalesced several notions of the quality of an ontology under the umbrella of the concept of a “beautiful ontology” others have introduced. In doing so we focussed on ontology evaluation criteria that are sufficiently well-defined to lead to repeatable assessments. Relying on published systematic literature reviews we selected four testing reference ontologies for assessment, namely STOWS, OntoTest, ROoST, and TestTDO.</div><div>Results indicate that a small number of published ontology assessment criteria have been defined with sufficient formality to allow their non-biased use. Nevertheless, we primarily based our assessment of the four selected ontologies on the necessary isomorphic mapping between an ontology and the domain it represents. Results indicate that none of the selected ontologies is designed with sufficient rigour. One observation we make is that published ontologies ought to be described more rigorously with a necessary complete dictionary describing all concepts, properties, relations and axioms.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112759"},"PeriodicalIF":4.1,"publicationDate":"2026-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145928351","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A reference architecture for ethical-aware autonomous systems","authors":"Marco Autili ,&nbsp;Martina De Sanctis ,&nbsp;Paola Inverardi ,&nbsp;Mashal Afzal Memon ,&nbsp;Patrizio Pelliccione ,&nbsp;Sara Pettinari","doi":"10.1016/j.jss.2025.112749","DOIUrl":"10.1016/j.jss.2025.112749","url":null,"abstract":"<div><div><em>Background.</em> Autonomous systems, whether AI-enabled or not, are ubiquitous and pervasive in our daily lives. While their adoption and use bring many benefits, they also pose significant ethical challenges. <em>Objective.</em> The objective of this work is to contribute a reference architecture for ethical-aware autonomous systems, focusing on their interaction and collaboration with humans, being them proactive, reactive or passive in the interaction with the systems. <em>Method.</em> To define the architecture, we analyzed scientific papers in the field, guidelines and recommendations, as well as laws and regulations. We then applied this acquired knowledge to build the reference architecture. The results of this work were validated through expert interviews and a scenario-based evaluation. <em>Results.</em> We contribute (i) a definition of ethical-aware autonomous systems, (ii) requirements for ethical-aware autonomous systems, and (iii) a reference architecture for ethical-aware autonomous systems. Our reference architecture is intended to help system and software engineers to design autonomous or intelligent systems that should interact and operate with humans in ethically sensitive contexts such as healthcare, social robotics, and assistive technologies. <em>Conclusion.</em> We believe that this work will assist software architects and engineers in designing and developing autonomous systems that should interact and collaborate with humans while respecting values important to individuals, society, and the environment.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112749"},"PeriodicalIF":4.1,"publicationDate":"2026-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145886520","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Clash: Enhancing context-sensitivity in data-flow analysis for mitigating the impact of indirect calls","authors":"Jinyan Xie,&nbsp;Yingzhou Zhang,&nbsp;Mingzhe Hu,&nbsp;Liping Han,&nbsp;Le Yu,&nbsp;Qiuran Ding","doi":"10.1016/j.jss.2025.112753","DOIUrl":"10.1016/j.jss.2025.112753","url":null,"abstract":"<div><div>Indirect calls pose significant challenges to static data-flow analysis due to their dynamic resolution at runtime, which introduces imprecision and inefficiency by allowing spurious data facts to propagate across mismatched call contexts. Prior work like Context-Free-Language (CFL) cannot maintain context-sensitivity in indirect call resolution, as the dynamically constructed data-flow edges lack sufficient context information. To address this limitation, we present <span>Clash</span>, a novel framework that enhances context-sensitivity in data-flow analysis without compromising scalability. <span>Clash</span> introduces two core innovations including Mutex identifiers and conflict detectors. Mutex identifiers statically approximate invocation contexts and are bound to function pointers to detect mismatches during data-flow analysis. Conflict detectors refer to Call-Store Constraint (CSC), Callback-Constraint (CBC), and Call-Return Constraint (CRC), which selectively suppress invalid inter-procedural data-flows at critical edges in the Static Value-Flow Graph (SVFG). These mechanisms enable <span>Clash</span> to efficiently prune infeasible data-flows while preserving necessary data dependencies.</div><div>Our evaluation shows that <span>Clash</span> achieves an average reduction of 76.93%, 51.79%, 53.97% and 48.23% in excessive data-flows compared to First-Layer Type Analysis (FLTA), Inter-procedural Finite Distributive Subset (IFDS/IDE), Static-Value-Flow (SVF) and Type-Flow Analysis (TFA) respectively. In terms of performance, <span>Clash</span> reduces whole-program analysis time by 82.83%, 27.79%, 41.4%, and 39.38% compared to these methods respectively. Therefore, <span>Clash</span> provides a scalable and precise solution for managing context-sensitive data-flows in the presence of complex indirect calls.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"235 ","pages":"Article 112753"},"PeriodicalIF":4.1,"publicationDate":"2026-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145928468","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}