Journal of Biomedical Informatics最新文献

{"title":"Medication information extraction using local large language models","authors":"Phillip Richter-Pechanski ,&nbsp;Marvin Seiferling ,&nbsp;Christina Kiriakou ,&nbsp;Dominic M. Schwab ,&nbsp;Nicolas A. Geis ,&nbsp;Christoph Dieterich ,&nbsp;Anette Frank","doi":"10.1016/j.jbi.2025.104898","DOIUrl":"10.1016/j.jbi.2025.104898","url":null,"abstract":"<div><h3>Objective</h3><div>Medication information is crucial for clinical routine and research. However, a vast amount is stored in unstructured text, such as doctor’s letters, requiring manual extraction – a resource-intensive, error-prone task. Automating this process comes with significant constraints in a clinical setup, including the demand for clinical expertise, limited time-resources, restricted IT infrastructure, and the demand for transparent predictions. Recent advances in generative large language models (LLMs) and parameter-efficient fine-tuning methods show potential to address these challenges.</div></div><div><h3>Methods</h3><div>We evaluated local LLMs for end-to-end extraction of medication information, combining named entity recognition and relation extraction. We used format-restricting instructions and developed an innovative feedback pipeline to facilitate automated evaluation. We applied token-level Shapley values to visualize and quantify token contributions, to improve transparency of model predictions.</div></div><div><h3>Results</h3><div>Two open-source LLMs – one general (Llama) and one domain-specific (OpenBioLLM) – were evaluated on the English n2c2 2018 corpus and the German CARDIO:DE corpus. OpenBioLLM frequently struggled with structured outputs and hallucinations. Fine-tuned Llama models achieved new state-of-the-art results, improving F1-score by up to 10 percentage points (pp.) for adverse drug events and 6 pp. for medication reasons on English data. On the German dataset, Llama established a new benchmark, outperforming traditional machine learning methods by up to 16 pp. micro average F1-score.</div></div><div><h3>Conclusion</h3><div>Our findings show that fine-tuned local open-source generative LLMs outperform SOTA methods for medication information extraction, delivering high performance with limited time and IT resources in a real-world clinical setup, and demonstrate their effectiveness on both English and German data. Applying Shapley values improved prediction transparency, supporting informed clinical decision-making.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"169 ","pages":"Article 104898"},"PeriodicalIF":4.5,"publicationDate":"2025-08-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144892008","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Resource-efficient instruction tuning of large language models for biomedical named entity recognition","authors":"Hui Liu ,&nbsp;Ziyi Chen ,&nbsp;Peilin Li ,&nbsp;Yuan-Zhi Liu ,&nbsp;Xiangtao Liu ,&nbsp;Ronald X. Xu ,&nbsp;Mingzhai Sun","doi":"10.1016/j.jbi.2025.104896","DOIUrl":"10.1016/j.jbi.2025.104896","url":null,"abstract":"<div><h3>Objective:</h3><div>Large language models (LLMs) have exhibited remarkable efficacy in natural language processing (NLP) tasks, with fine-tuning for Biomedical Named Entity Recognition (BioNER) receiving significant research attention. However, the substantial computational demands associated with fine-tuning large-scale models constrain their development and deployment. Consequently, this study investigates parameter-efficient fine-tuning (PEFT) techniques to optimize LLMs for BioNER under limited computational resources. By leveraging these methods, competitive model performance is maintained while preserving in-domain generalization capability.</div></div><div><h3>Methods:</h3><div>In this study, we employed the PEFT method QLoRA to fine-tune the open-source Llama3.1 model, developing the NERLlama3.1 model specifically designed for the BioNER task. First, an LLM instruction tuning dataset was created using BioNER datasets such as NCBI-disease, BC5CDR-chem, and BC2GM-gene. Next, the Llama3.1-8B model was fine-tuned using the QLoRA method on a single 16GB memory GPU. Furthermore, during the inference phase, we introduced a prompt engineering technique called self-consistency NER prompting (SCNP). This approach leverages the diversity of outputs generated by LLMs to significantly enhance NER performance. Finally, we also developed a multi-task BioNER-capable model, NERLlama3.1-MT, to investigate the capability of fine-tuned LLMs in addressing multi-task BioNER scenarios.</div></div><div><h3>Results:</h3><div>The NERLlama3.1 model achieved F1-scores of 0.8977, 0.9402, and 0.8530 on the NCBI-disease, BC5CDR-chemical, and BG2GM-gene datasets, respectively. Furthermore, when evaluated on previously unseen datasets, it attained F1-scores of 0.6867 on BC5CDR-disease, 0.6800 on NLM-chemical, and 0.8378 on NLM-gene. These results demonstrate that NERLlama3.1 not only outperforms fully fine-tuned LLMs but also exhibits superior in-domain generalization capabilities when compared to the BERT-base model. Additionally, this work represents the first exploration of fine-tuning LLMs for multi-task BioNER.</div></div><div><h3>Conclusion:</h3><div>NERLlama3.1 outperformed LLMs fine-tuned with full parameter updates, despite requiring significantly fewer computational resources. Moreover, it exhibited substantially superior in-domain generalization capabilities compared to traditional pre-trained language models. Its low resource demands, high performance, and strong generalization enhance its applicability and utility across diverse clinical BioNER tasks.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"170 ","pages":"Article 104896"},"PeriodicalIF":4.5,"publicationDate":"2025-08-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144926248","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Biomedical knowledge graph verification with multitask learning architectures","authors":"Chih-Ping Wei ,&nbsp;Pei-Yuan Tsai ,&nbsp;Jih-Jane Li","doi":"10.1016/j.jbi.2025.104894","DOIUrl":"10.1016/j.jbi.2025.104894","url":null,"abstract":"<div><h3>Objective</h3><div>Large-scale biomedical KGs, typically constructed using automated entity-relation extraction methods from vast amounts of textual documents, often contain erroneous biomedical triplets, which raises concerns about their quality. Using such noisy KGs in downstream applications can compromise the validity of biomedical research and lead to inaccurate conclusions. This study aims to design an effective knowledge graph verification (KGV) method to determine the correctness of triplets in biomedical KGs, enabling the removal of erroneous triplets identified through the proposed method.</div></div><div><h3>Methods</h3><div>We propose a multitask-learning-based KGV (referred to as the MTL-KGV) method, which includes two key stages: 1) KG embedding (KGE) learning and (2) triplet classification model learning. In addition, we explore three types of multitask learning (MTL) architectures—hard parameter sharing (HPS), multi-gate mixture-of-experts (MMoE), and customized gate control (CGC)—for triplet classification model learning.</div></div><div><h3>Results</h3><div>Using SemMedDB as a data source to construct a large-scale KG for KGE training and a dataset of 6,427 biomedical triplets annotated by a domain expert, we empirically evaluate the effectiveness of our proposed MTL-KGV method by comparing it to several benchmark methods. Our evaluation results indicate that all three versions of our proposed MTL-KGV method consistently outperform the benchmark methods. Moreover, our proposed method with the MMoE multitask learning architecture emerges as the most effective for detecting erroneous biomedical triplets.</div></div><div><h3>Conclusion</h3><div>This work contributes to KGV research by introducing a multitask learning framework tailored for KGV. The proposed MTL-KGV method improves the quality of biomedical KGs, thereby supporting downstream applications and advancing biomedical research that relies on these biomedical KGs.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"169 ","pages":"Article 104894"},"PeriodicalIF":4.5,"publicationDate":"2025-08-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144892060","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Multimodal large language models for medical image diagnosis: Challenges and opportunities","authors":"Andrew Zhang,&nbsp;Eric Zhao,&nbsp;Ruirui Wang,&nbsp;Xiuqi Zhang,&nbsp;Justin Wang,&nbsp;Ethan Chen","doi":"10.1016/j.jbi.2025.104895","DOIUrl":"10.1016/j.jbi.2025.104895","url":null,"abstract":"<div><div>The integration of artificial intelligence (AI) into radiology has significantly improved diagnostic accuracy and workflow efficiency. Multimodal large language models (MLLMs), which combine natural language processing (NLP) and computer vision techniques, hold the potential to further revolutionize medical image analysis. Despite these advances, their widespread clinical adoption of MLLMs remains limited by challenges such as data quality, interpretability, ethical and regulatory compliance- including adherence to frameworks like the General Data Protection Regulation (GDPR) − computational demands, and generalizability across diverse patient populations. Addressing these interconnected challenges presents opportunities to enhance MLLM performance and reliability. Priorities for future research include improving model transparency, safeguarding data privacy through federated learning, optimizing multimodal fusion strategies, and establishing standardized evaluation frameworks. By overcoming these barriers, MLLMs can become essential tools in radiology, supporting clinical decision-making, and improving patient outcomes.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"169 ","pages":"Article 104895"},"PeriodicalIF":4.5,"publicationDate":"2025-08-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144885693","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"AI-driven approach for creating and evaluating a synthetic dataset for Medication Errors","authors":"Hanae Touati ,&nbsp;Rafika Thabet ,&nbsp;Franck Fontanili ,&nbsp;Marie-Hélène Cleostrate ,&nbsp;Marc Pruski ,&nbsp;Marie-Noëlle Cufi ,&nbsp;Elyes Lamine","doi":"10.1016/j.jbi.2025.104889","DOIUrl":"10.1016/j.jbi.2025.104889","url":null,"abstract":"<div><h3>Objective:</h3><div>This study aims to create a complete Medication Error (ME) dataset. This will help to address the challenge of limited access to real-world data for developing machine learning models in healthcare applications.</div></div><div><h3>Methods:</h3><div>We use transformer-based models (GPT-4, LLAMA3, and Mistral) to create our synthetic dataset in French. These models generate a diverse range of descriptions that capture the variability of ME types. We assess the effectiveness of our synthetic dataset through expert evaluations by healthcare professionals and an AI-driven analysis, to test its realism and its utility in training machine learning models for ME classification.</div></div><div><h3>Results:</h3><div>The synthetic dataset demonstrates high accuracy and realism in representing diverse ME scenarios. Expert evaluation confirms that the dataset is similar to real-world ME data. The AI-driven evaluation also shows that models trained on synthetic data achieved robust classification performance, validating the dataset’s utility for the development of effective ME classification tools.</div></div><div><h3>Conclusion:</h3><div>The proposed approach demonstrates the potential of large language models to generate realistic synthetic ME reports in French. Out of 200 evaluated reports, 70% of zero-shot outputs were deemed below expectations, while 80% of one-shot and few-shot outputs were considered valid or valid with minor revisions by clinical experts. Furthermore, classifiers trained on 800 synthetic reports attained an F1-score of up to 0.78 when tested on real data. These results confirm that synthetic data can effectively support AI-driven ME analysis in contexts where real-world data is limited or unavailable.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"169 ","pages":"Article 104889"},"PeriodicalIF":4.5,"publicationDate":"2025-08-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144835215","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Rapid review: Growing usage of Multimodal Large Language Models in healthcare","authors":"Pallavi Gupta ,&nbsp;Zhihong Zhang ,&nbsp;Meijia Song ,&nbsp;Martin Michalowski ,&nbsp;Xiao Hu ,&nbsp;Gregor Stiglic ,&nbsp;Maxim Topaz","doi":"10.1016/j.jbi.2025.104875","DOIUrl":"10.1016/j.jbi.2025.104875","url":null,"abstract":"<div><h3>Objective:</h3><div>Recent advancements in large language models (LLMs) have led to multimodal LLMs (MLLMs), which integrate multiple data modalities beyond text. Although MLLMs show promise, there is a gap in the literature that empirically demonstrates their impact in healthcare. This paper summarizes the applications of MLLMs in healthcare, highlighting their potential to transform health practices.</div></div><div><h3>Methods:</h3><div>A rapid literature review was conducted in August 2024 using World Health Organization (WHO) rapid-review methodology and PRISMA standards, with searches across four databases (Scopus, Medline, PubMed and ACM Digital Library) and top-tier conferences—including NeurIPS, ICML, AAAI, MICCAI, CVPR, ACL and EMNLP. Articles on MLLMs healthcare applications were included for analysis based on inclusion and exclusion criteria.</div></div><div><h3>Results:</h3><div>The search yielded 115 articles, 39 included in the final analysis. Of these, 77% appeared online (preprints and published) in 2024, reflecting the emergence of MLLMs. 80% of studies were from Asia and North America (mainly China and US), with Europe lagging. Studies split evenly between pre-built MLLMs evaluations (60% focused on GPT versions) and custom MLLMs/frameworks development with task-specific customizations. About 81% of studies examined MLLMs for diagnosis and reporting in radiology, pathology, and ophthalmology, with additional applications in education, surgery, and mental health. Prompting strategies, used in 80% of studies, improved performance in nearly half. However, evaluation practices were inconsistent with 67% reported accuracy. Error analysis was mostly anecdotal, with only 18% categorized failure types. Only 13% validated explainability through clinician feedback. Clinical deployment was demonstrated in just 3% of studies, and workflow integration, governance, and safety were rarely addressed.</div></div><div><h3>Discussion and Conclusion:</h3><div>MLLMs offer substantial potential for healthcare transformation through multimodal data integration. Yet, methodological inconsistencies, limited validation, and underdeveloped deployment strategies highlight the need for standardized evaluation metrics, structured error analysis, and human-centered design to support safe, scalable, and trustworthy clinical adoption.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"169 ","pages":"Article 104875"},"PeriodicalIF":4.5,"publicationDate":"2025-08-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144773210","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Measuring disease burden with individual cumulative incidence in patients with cirrhosis","authors":"Mitchell Paukner ,&nbsp;Daniela P. Ladner ,&nbsp;CAPriCORN Team,&nbsp;Lihui Zhao","doi":"10.1016/j.jbi.2025.104883","DOIUrl":"10.1016/j.jbi.2025.104883","url":null,"abstract":"<div><h3>Objective:</h3><div>Introduce a new method for evaluating the health history of a patient through the use of multi-type recurrent events data that can aid in the assessment of disease progression and quality of life in a healthcare setting.</div></div><div><h3>Methods:</h3><div>The Disease Burden Score (DBS) is characterized as the area under the Disease Burden Curve (DBC), a monotone, increasing stepwise graph, that measures the weighted or unweighted number of health events that occur during a patient’s follow-up period (i.e. cardiovascular events or decompensation events).</div></div><div><h3>Results:</h3><div>The performance of our method, evaluated in simulation studies, demonstrated that our modeling method produces unbiased results and can improve power over alternatives in common biomedical research settings. The method was also applied to real data from a collection of Electronic Health Records.</div></div><div><h3>Conclusion:</h3><div>A DBS can be computed for all patients present in the data and thus can be used as a means of comparing subgroups and as the outcome variable in regression. This measure is not only a valuable tool in cases where death data is not available or reliable but also as an interim measurement when death is infrequent.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"169 ","pages":"Article 104883"},"PeriodicalIF":4.5,"publicationDate":"2025-07-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144768695","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"CLEAR: A vision to support clinical evidence lifecycle with continuous learning","authors":"Yilu Fang ,&nbsp;Gongbo Zhang ,&nbsp;Fangyi Chen ,&nbsp;George Hripcsak ,&nbsp;Yifan Peng ,&nbsp;Patrick Ryan ,&nbsp;Chunhua Weng","doi":"10.1016/j.jbi.2025.104884","DOIUrl":"10.1016/j.jbi.2025.104884","url":null,"abstract":"<div><div>Human knowledge of diseases, treatments, and prevention techniques is constantly evolving. The generation of clinical evidence using randomized controlled trials on human subjects occurs notably slowly and inefficiently. The Learning Health System (LHS) has been proposed to facilitate the continuous improvement of individual and population health through a cycle of knowledge, practice, and data. However, the gap between the demand for high-quality evidence to support clinical decisions and the available evidence continues to enlarge. While the current LHS vision articulates the integration of Real-World Data (RWD), the rapid generation of RWD often outpaces the rate of effective evidence synthesis and implementation. Considering this, we propose a new framework that more effectively leverages RWD to support the entire clinical evidence lifecycle through a continuous learning mechanism. This framework, powered by modern data science and informatics, offers enhanced scalability and efficiency. In this vision, specifically, RWD is integrated into the clinical evidence lifecycle via four closed feedback loops: 1) guiding research prioritization and study design, 2) facilitating clinical guideline development, 3) assisting guideline evaluation, and 4) supporting shared decision-making. Our framework enables rapid responsiveness to emerging health data and evolving healthcare needs, timely development of clinical guidelines to optimize clinical recommendations, and sustained improvements in clinical practice and patient outcomes. This vision calls for informatics support for an efficient, scalable, and stakeholder-aware clinical evidence lifecycle.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"169 ","pages":"Article 104884"},"PeriodicalIF":4.5,"publicationDate":"2025-07-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144760223","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"An open-set semi-supervised multi-task learning framework for context classification in biomedical texts","authors":"Difei Tang ,&nbsp;Thomas Yu Chow Tam ,&nbsp;Haomiao Luo ,&nbsp;Cheryl A. Telmer ,&nbsp;Natasa Miskov-Zivanov","doi":"10.1016/j.jbi.2025.104886","DOIUrl":"10.1016/j.jbi.2025.104886","url":null,"abstract":"<div><h3>Objective</h3><div>In biomedical research, knowledge about the relationships between entities, including genes, proteins, and drugs, is vital for elucidating complex biological processes and intracellular pathway mechanisms. While natural language processing (NLP) methods have shown great success in biomedical relation extraction (RE), extracted relations often lack contextual information such as cell type, cell line, and intracellular location. Previous studies treated this problem as a post hoc context-relation association task, limited by the absence of a golden standard corpus and prone to error propagation. To address these challenges, we propose CELESTA (Context Extraction through LEarning with Semi-supervised multi-Task Architecture), an open-set semi-supervised multi-task learning (OSSL-MTL) framework for biomedical context classification.</div></div><div><h3>Methods</h3><div>We designed a multi-task learning (MTL) architecture that integrates with the semi-supervised learning (SSL) strategies to leverage unlabeled data containing both in-distribution (ID) and out-of-distribution (OOD) examples. We created a large-scale dataset consisting of five context classification tasks by curating two large Biological Expression Language (BEL) corpora and annotating them with our new entity span annotation method. Additionally, we developed an OOD detector to distinguish between ID and OOD instances within the unlabeled data and applied data augmentation with an external database to enrich our dataset.</div></div><div><h3>Results</h3><div>Extensive experiments show that our framework significantly improves context classification performance. Our best OSSL-MTL models achieve F1 scores of 77.75% and 82.87% on location and disease classification tasks, and the SSL-MTL models without OOD detection perform best for cell line and cell type classification. The OOD detection experiment confirms that the OOD detector effectively identifies unknown categories while maintaining ID accuracy. Qualitative analysis shows improved extraction of implicit contexts compared to baseline models.</div></div><div><h3>Conclusion</h3><div>Our analysis demonstrates the effectiveness of the framework CELESTA in improving context classification and extracting contextual information with high accuracy. The newly created dataset and code are publicly available on GitHub (<span><span>https://github.com/pitt-miskov-zivanov-lab/CELESTA</span><svg><path></path></svg></span>).</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"169 ","pages":"Article 104886"},"PeriodicalIF":4.5,"publicationDate":"2025-07-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144742203","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Conceptual framework for prediction models of patient deterioration based on nursing documentation patterns: reproducibility and generalizability with a large number of hospitals across the United States","authors":"Yik-Ki Jacob Wan ,&nbsp;Samir E. Abdelrahman ,&nbsp;Julio C. Facelli ,&nbsp;Karl Madaras-Kelly ,&nbsp;Kensaku Kawamoto ,&nbsp;Deniz Dishman ,&nbsp;Samuel R. Himes ,&nbsp;Kenrick Cato ,&nbsp;Sarah C. Rossetti ,&nbsp;Guilherme Del Fiol","doi":"10.1016/j.jbi.2025.104887","DOIUrl":"10.1016/j.jbi.2025.104887","url":null,"abstract":"<div><h3>Objective</h3><div>The Health Process Model (HPM)-ExpertSignals Conceptual Framework posits that healthcare professionals’ patient care behaviors can be used to predict in-hospital deterioration. Prediction models based on this framework have been validated using data from 4 hospitals within two healthcare systems. As clinician-system interactions may differ across organizations, this study aimed to evaluate the reproducibility and generalizability of the underlying conceptual framework using data from over 200 hospitals across the US.</div></div><div><h3>Methods</h3><div>This study used eICU-CRD, a publicly accessible dataset with data from 208 US hospitals. A logistic regression model was developed to predict in-hospital deterioration following the HPM-ExpertSignals conceptual framework. To test its reproducibility, patients were randomly split into training and testing datasets. After bootstrap testing of the model, the mean precision-recall curve (AUPRC) was compared with outcomes from previously published studies. For generalizability testing, the hospitals in the dataset were randomly assigned into model training or testing sets. After the model was trained with training hospitals’ data, generalizability was measured as the percentage of testing hospitals with an AUPRC at or above a baseline performance obtained in the reproducibility experiment.</div></div><div><h3>Results</h3><div>The AUPRC in the reproducibility experiment was 0.10 (0.09,0.11; 95% CI), equivalent to the AUPRC reported in a previous study at 0.093 (0.09, 0.096; 95% CI). In the generalizability experiment, 94% of the testing hospitals had AUPRC at or above the baseline AUPRC of 0.10.</div></div><div><h3>Conclusion</h3><div>The study provides evidence supporting the reproducibility of a predictive model following the HPM-ExpertSignals framework. This model also generalized to most hospitals without additional training. Nevertheless, some hospitals still obtained lower-than-expected performance, highlighting the need for model evaluation and potential fine-tuning before local adoption. Similar studies are needed to investigate the reproducibility and generalizability of other classes of machine learning models in healthcare.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"169 ","pages":"Article 104887"},"PeriodicalIF":4.5,"publicationDate":"2025-07-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144724331","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}