JAMIA Open最新文献

{"title":"Predicting falls using electronic health records: a time series approach.","authors":"Peter J Hoover, Terri L Blumke, Anna D Ware, Malvika Pillai, Zachary P Veigulis, Catherine M Curtin, Thomas F Osborne","doi":"10.1093/jamiaopen/ooaf116","DOIUrl":"10.1093/jamiaopen/ooaf116","url":null,"abstract":"<p><strong>Objective: </strong>To develop a more accurate fall prediction model within the Veterans Health Administration.</p><p><strong>Materials and methods: </strong>The cohort included Veterans admitted to a Veterans Health Administration acute care setting from July 1, 2020, to June 30, 2022, with a length of stay between 1 and 7 days. Demographic and clinical data were obtained through electronic health records. Veterans were identified as having a documented fall through clinical progress notes. A transformer model was used to obtain features of this data, which was then used to train a Light Gradient-Boosting Machine for classification and prediction. Area under the precision-recall curve assisted in model tuning, with geometric mean used to define an optimal classification threshold.</p><p><strong>Results: </strong>Among 242,844 Veterans assessed, 5965 (2.5%) were documented as having a fall during their clinical stay. Employing a transformer model with a Light Gradient-Boosting Machine resulted in an area under the curve of .851 and an area under the precision-recall curve of .285. With an accuracy of 76.3%, the model resulted in a specificity of 76.2% and a sensitivity of 77.3%.</p><p><strong>Discussion: </strong>Prior evaluations have highlighted limitations of the Morse Fall Scale (MFS) in accurately assessing fall risk. Developing a time series classification model using existing electronic health record data, our model outperformed traditional MFS-based evaluations and other fall-risk models. Future work is necessary to address limitations, including class imbalance and the need for prospective validation.</p><p><strong>Conclusion: </strong>An improvement over the MFS, this model, automatically calculated from existing data, can provide a more efficient and accurate means for identifying patients at risk of fall.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"8 5","pages":"ooaf116"},"PeriodicalIF":3.4,"publicationDate":"2025-10-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12492486/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145233301","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Analyzing the impact of pharmacogenomics-guided nonsteroidal anti-inflammatory drug alerts in clinical practice.","authors":"Amanda Massmann, Natasha J Petry, Max Weaver, Halle Brady, Roxana A Lupu","doi":"10.1093/jamiaopen/ooaf112","DOIUrl":"10.1093/jamiaopen/ooaf112","url":null,"abstract":"<p><strong>Objectives: </strong>This study evaluates response rates of pharmacogenomics (PGx) nonsteroidal anti-inflammatory drugs (NSAIDs) clinical decision support (CDS) alerts at Sanford Health from May 2020 to December 2024.</p><p><strong>Materials and methods: </strong>A retrospective analysis was conducted on PGx NSAIDs interruptive alerts. Response options were classified into five categories (1) continuation of triggering NSAID order, (2) dose modification, (3) alternative NSAID ordered without PGx implications, (4) alternative analgesic (ie, opioid) ordered, and (5) discontinuation of NSAID without alternative therapy.</p><p><strong>Results: </strong>The study analyzed 2361 alert instances from 978 patients. The most common response was discontinuing NSAID without alternative therapy (43%). Dose modifications and orders for alternative analgesics comprised 2.57% and 14.67% of responses, respectively. The initial acceptance rate was 62.6%. Prior NSAID use significantly impacted override rates (60% vs 40%, <i>P</i> < .001). A 409-day breaking point was observed to affect alert acceptance rates, with the highest acceptance in NSAID naïve patients (96.1%).</p><p><strong>Discussion: </strong>PGx NSAIDs CDS alert acceptance rates were higher compared to general CDS acceptance rates. This study highlights opportunities for continuous improvement including optimizing alert modality, modifying alert criteria to include look-back periods, and implementing genetically adapted ordersets.</p><p><strong>Conclusion: </strong>The initial acceptance rate of PGx NSAIDs CDS alerts was 62.6%, however, with significantly higher acceptance rates in NSAID naïve patients (62.6% vs 96.1%, <i>P</i> < .001). Integration of CDS is vital to the successful implementation of PGx in clinical practice.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"8 5","pages":"ooaf112"},"PeriodicalIF":3.4,"publicationDate":"2025-10-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12492481/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145233287","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Opportunities, barriers, and remedies for implementing REDCap integration with electronic health records via Fast Healthcare Interoperability Resources (FHIR).","authors":"Alex C Cheng, Cathy Shyr, Adam Lewis, Francesco Delacqua, Teresa Bosler, Mary K Banasiewicz, Robert Taylor, Christopher J Lindsell, Paul A Harris","doi":"10.1093/jamiaopen/ooaf111","DOIUrl":"10.1093/jamiaopen/ooaf111","url":null,"abstract":"<p><strong>Objective: </strong>Accelerate adoption of clinical research technology that obtains electronic health record (EHR) data through HL7 Fast Healthcare Interoperability Resources (FHIR).</p><p><strong>Materials and methods: </strong>Based on experience helping institutions implement REDCap-EHR integration and surveys of users and potential users, we discuss the technical and organizational barriers to adoption with strategies for remediation.</p><p><strong>Results: </strong>With strong demand from researchers, the 21st Century Cures Act Final Rule in place, and REDCap software already in use at most research organizations, the environment seems ideal for REDCap-EHR integration for automated data exchange. However, concerns from information technology and regulatory leaders often slow progress and restrict how and when data from the EHR can be used.</p><p><strong>Discussion and conclusion: </strong>While technological controls can help alleviate concerns about FHIR applications used in research, we have found that messaging, education, and extramural funding remain the strongest drivers of adoption.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"8 5","pages":"ooaf111"},"PeriodicalIF":3.4,"publicationDate":"2025-10-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12494410/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145233256","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Synthetic data for pharmacogenetics: enabling scalable and secure research.","authors":"Marko Miletic, Anna Bollinger, Samuel S Allemann, Murat Sariyar","doi":"10.1093/jamiaopen/ooaf107","DOIUrl":"10.1093/jamiaopen/ooaf107","url":null,"abstract":"<p><strong>Objective: </strong>This study evaluates the performance of 7 synthetic data generation (SDG) methods-synthpop, avatar, copula, copulagan, ctgan, tvae, and the large language models-based tabula-for supporting pharmacogenetics (PGx) research.</p><p><strong>Materials and methods: </strong>We used PGx profiles from 142 patients with adverse drug reactions or therapeutic failures, considering 2 scenarios: (1) a high-dimensional genotype dataset (104 variables) and (2) a phenotype dataset (24 variables). Models were assessed for (1) broad utility using propensity score mean squared error ( <math><mi>pMSE</mi></math> ), (2) specific utility via weighted <math> <mrow> <msub><mrow><mi>F</mi></mrow> <mrow><mn>1</mn></mrow> </msub> </mrow> </math> score in a Train-Synthetic-Test-Real framework, and (3) privacy risk as ε-identifiability.</p><p><strong>Results: </strong>Copula and synthpop consistently achieved strong performance across both datasets, combining low ε-identifiability (0.25-0.35) with competitive utility. Deep learning models like tabula and tvae trained for 10 000 epochs achieved lower <math><mi>pMSE</mi></math> but had higher ε-identifiability (>0.4) and limited gains in predictive performance. Specific utility was only weakly correlated with broad utility, indicating that distributional fidelity does not ensure predictive relevance. Copula and synthpop often outperformed original data in weighted <math> <mrow> <msub><mrow><mi>F</mi></mrow> <mrow><mn>1</mn></mrow> </msub> </mrow> </math> scores, especially under noise or data imbalance.</p><p><strong>Discussion: </strong>While deep learning models can achieve high distributional fidelity ( <math><mi>pMSE</mi></math> ), they often incur elevated ε-identifiability, raising privacy concerns. Traditional methods like copula and synthpop consistently offer robust utility and lower re-identification risk, particularly for high-dimensional data. Importantly, general utility does not predict specific utility ( <math> <mrow> <msub><mrow><mi>F</mi></mrow> <mrow><mn>1</mn></mrow> </msub> </mrow> </math> score), emphasizing the need for multimetric evaluation.</p><p><strong>Conclusion: </strong>No single SDG method dominated across all criteria. For privacy-sensitive PGx applications, classical methods such as copula and synthpop offer a reliable trade-off between utility and privacy, making them preferable for high-dimensional, limited-sample settings.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"8 5","pages":"ooaf107"},"PeriodicalIF":3.4,"publicationDate":"2025-10-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12492482/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145233431","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Adding employment status in the medical record demonstrates its importance as a social determinant of health.","authors":"Laura E Breeher, Samantha Westphal, Tammy Green, Alzhraa Abbas, Clayton T Cowl","doi":"10.1093/jamiaopen/ooaf108","DOIUrl":"10.1093/jamiaopen/ooaf108","url":null,"abstract":"<p><strong>Objective: </strong>To share findings of a quality improvement initiative to capture employment status in the electronic medical record (EMR) to mitigate the potential impact of work loss on the health of patients by utilizing the results to identify eligible individuals for resources to assist return-to-work efforts.</p><p><strong>Materials and methods: </strong>Patients self-identified employment status through a structured new social determinants of health (SDOH) question within the EMR. An electronic outreach campaign was developed to provide information via the patient portal detailing services within healthcare and the community that could benefit the patients.</p><p><strong>Results: </strong>Over the course of 12 months, 2059 patients were identified from the employment SDOH question. Resources to support stay at work and return to work efforts were provided to patients through an automated electronic portal campaign with 87% of patients reading the message and 7% engaging with a healthcare return to work case manager.</p><p><strong>Discussion: </strong>Loss of employment has detrimental impacts on individual and population health. Most EMRs do not capture information on employment status. Adding this simple question identified individuals with potential gaps in SDOH, and in this case allowed specific resources to be shared with patients with an illness or injury that was acutely impacting work.</p><p><strong>Conclusion: </strong>Medical center decision makers and EMR programmers should consider adding employment status as a social determinant of health.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"8 5","pages":"ooaf108"},"PeriodicalIF":3.4,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12488229/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145213795","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Workflows to automate covariate-adaptive randomization in REDCap via data entry triggers.","authors":"Jacob M Schauer, Marc O Broxton, Luke V Rasmussen, Gregory Swann, Michael E Newcomb, Jody D Ciolino","doi":"10.1093/jamiaopen/ooaf110","DOIUrl":"10.1093/jamiaopen/ooaf110","url":null,"abstract":"<p><strong>Objective: </strong>Covariate-adaptive randomization algorithms (CARAs) can reduce covariate imbalance in randomized controlled trials (RCTs), but a lack of integration into Research Electronic Data Capture (REDCap) has limited their use. We developed a software pipeline to seamlessly integrate CARAs into REDCap as part of the all2GETHER study, a 2-armed RCT concerning HIV prevention.</p><p><strong>Materials and methods: </strong>Leveraging REDCap's Data Entry Trigger and a separate server, we implemented software in PHP and R to automate randomizations for all2GETHER. Randomizations were triggered by saving a specific REDCap form and were automatically communicated to unblinded study personnel.</p><p><strong>Results: </strong>Study arms were highly comparable, with differences across covariates characterized by Cohen's <i>d</i> = 0.003 for continuous variables and risk differences <2.4% for categorical/binary variables.</p><p><strong>Conclusions: </strong>Our pipeline proved effective at reducing covariate imbalance with minimal additional effort for study personnel.</p><p><strong>Discussion: </strong>This pipeline is reproducible and could be used by other RCTs that collect data via REDCap.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"8 5","pages":"ooaf110"},"PeriodicalIF":3.4,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12486239/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145213994","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Leveraging open-source large language models for clinical information extraction in resource-constrained settings.","authors":"Luc Builtjes, Joeran Bosma, Mathias Prokop, Bram van Ginneken, Alessa Hering","doi":"10.1093/jamiaopen/ooaf109","DOIUrl":"10.1093/jamiaopen/ooaf109","url":null,"abstract":"<p><strong>Objective: </strong>We aimed to evaluate the zero-shot performance of open-source generative large language models (LLMs) on clinical information extraction from Dutch medical reports using the Diagnostic Report Analysis: General Optimization of NLP (DRAGON) benchmark.</p><p><strong>Methods: </strong>We developed and released the llm_extractinator framework, a scalable, open-source tool for automating information extraction from clinical texts using LLMs. We evaluated 9 multilingual open-source LLMs across 28 tasks in the DRAGON benchmark, covering classification, regression, and named entity recognition (NER). All tasks were performed in a zero-shot setting. Model performance was quantified using task-specific metrics and aggregated into a DRAGON utility score. Additionally, we investigated the effect of in-context translation to English.</p><p><strong>Results: </strong>Llama-3.3-70B achieved the highest utility score (0.760), followed by Phi-4-14B (0.751), Qwen-2.5-14B (0.748), and DeepSeek-R1-14B (0.744). These models outperformed or matched a fine-tuned RoBERTa baseline on 17 of 28 tasks, particularly in regression and structured classification. NER performance was consistently low across all models. Translation to English consistently reduced performance.</p><p><strong>Discussion: </strong>Generative LLMs demonstrated strong zero-shot capabilities on clinical natural language processing tasks involving structured inference. Models around 14B parameters performed well overall, with Llama-3.3-70B leading but at high computational cost. Generative models excelled in regression tasks, but were hindered by token-level output formats for NER. Translation to English reduced performance, emphasizing the need for native language support.</p><p><strong>Conclusion: </strong>Open-source generative LLMs provide a viable zero-shot alternative for clinical information extraction from Dutch medical texts, particularly in low-resource and multilingual settings.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"8 5","pages":"ooaf109"},"PeriodicalIF":3.4,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12488231/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145213796","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Early auxiliary diagnosis model for chest pain triad based on artificial intelligence multimodal fusion.","authors":"Jun Tang, Fang Chen, Dongdong Wu","doi":"10.1093/jamiaopen/ooaf114","DOIUrl":"10.1093/jamiaopen/ooaf114","url":null,"abstract":"<p><strong>Objectives: </strong>Acute chest pain is a common presentation in the emergency department, characterized by sudden onset with high morbidity and mortality. Traditional diagnostic methods, such as computed tomography (CT) and CT angiography (CTA), are often time-consuming and fail to meet the urgent need for rapid triage in emergency settings.</p><p><strong>Materials and methods: </strong>We developed a multimodal model that integrates Bio-ClinicalBERT and ensemble learning (AdaBoost, Gradient boosting, and XGBoost) based on 41 382 patient data from April 1, 2013 to April 1, 2025 at Chongqing Daping Hospital. By integrating clinical texts and laboratory indicators, the model aims to classify the 3 major causes of fatal chest pain (acute coronary syndrome, pulmonary embolism, and aortic dissection), as well as other causes of chest pain, aiding rapid triage. We adopt strict data preprocessing and rank importance feature selection.</p><p><strong>Results: </strong>The multimodal fusion model based on Gradient boosting exhibits the best performance: accuracy of 88.40%, area under the curve of 0.951, F1-score of 74.56%, precision of 77.50%, and recall of 72.52%. SHapley Additive exPlanations (SHAP) analysis confirmed the clinical relevance of key features such as d-dimer and high-sensitivity troponin. When reducing the number of numerical features to 30 key indicators, the model enhanced robustness without compromising performance.</p><p><strong>Discussion and conclusion: </strong>We developed an artificial intelligence model for chest pain classification that effectively addresses the problem of overlapping clinical symptoms through multimodal fusion, and the model has high accuracy. However, future work needs to better integrate model development with clinical workflows and practical constraints.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"8 5","pages":"ooaf114"},"PeriodicalIF":3.4,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12486235/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145213844","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Automated phenotyping of congenital heart disease for dynamic patient aggregation and outcome reporting.","authors":"Shuhei Toba, Taylor M Smith, Francesca Sperotto, Chrystalle Katte Carreon, Kwannapas Saengsin, Samuel Casella, Marlon Delgado, Peng Zeng, Stephen P Sanders, Audrey Dionne, Eric N Feins, Steven D Colan, John E Mayer, John N Kheir","doi":"10.1093/jamiaopen/ooaf106","DOIUrl":"10.1093/jamiaopen/ooaf106","url":null,"abstract":"<p><strong>Objectives: </strong>Accurate characterization of patients with congenital heart disease is fundamental to research, outcomes reporting, quality improvement, and clinical decision-making. Here we present an approach to computing the anatomy of patients with congenital heart disease based on the whole of their diagnostic and surgical codes.</p><p><strong>Materials and methods: </strong>All diagnostic and procedure codes for patients cared for between 1981 and 2020 at Boston Children's Hospital were extracted from a database containing diagnostic codes from echocardiograms, and procedural codes from surgical and catheterization procedures. The pipeline sequentially (1) mapped each of the 7500 native codes to algorithm codes; (2) computed the parent anatomy for each study using a pre-defined hierarchy; (3) computed the parent anatomy for the patient, based on highest ranking parent anatomy; and (4) computed the subcategories and mandatory co-variate findings for each patient. Thereafter, diagnostic accuracy of 500 unseen patients was adjudicated against clinical documentation by clinical experts.</p><p><strong>Results: </strong>A total of 514 541 echocardiograms on 161 735 patients were available for this study. Phenotypes of congenital cardiac diseases were assigned in 84 285 patients (52%), and the remainder were computed to have normal anatomy. Clinicians agreed with algorithm assignments in 96.4% (482 of 500 patients), with disagreements most often representing definitional differences. An interactive dashboard enabled by the output of this algorithm is presented.</p><p><strong>Conclusions: </strong>The computation of detailed congenital heart defect phenotypes from raw diagnostic and procedure codes is possible with a high degree of accuracy and efficiency. This framework may enable tools to support interactive outcomes reporting and clinical decision support.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"8 5","pages":"ooaf106"},"PeriodicalIF":3.4,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12486236/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145213774","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Evaluating the impact of data biases on algorithmic fairness and clinical utility of machine learning models for prolonged opioid use prediction.","authors":"Behzad Naderalvojoud, Catherine Curtin, Steven M Asch, Keith Humphreys, Tina Hernandez-Boussard","doi":"10.1093/jamiaopen/ooaf115","DOIUrl":"10.1093/jamiaopen/ooaf115","url":null,"abstract":"<p><strong>Objectives: </strong>The growing use of machine learning (ML) in healthcare raises concerns about how data biases affect real-world model performance. While existing frameworks evaluate algorithmic fairness, they often overlook the impact of bias on generalizability and clinical utility, which are critical for safe deployment. Building on prior methods, this study extends bias analysis to include clinical utility, addressing a key gap between fairness evaluation and decision-making.</p><p><strong>Materials and methods: </strong>We applied a 3-phase evaluation to a previously developed model predicting prolonged opioid use (POU), validated on Veterans Health Administration (VHA) data. The analysis included internal and external validation, model retraining on VHA data, and subgroup evaluation across demographic, vulnerable, risk, and comorbidity groups. We assessed performance using area under the receiver operating characteristic curve (AUROC), calibration, and decision curve analysis, incorporating standardized net-benefits to evaluate clinical utility alongside fairness and generalizability.</p><p><strong>Results: </strong>The internal cohort (<i>N</i> = 41 929) had a 14.7% POU prevalence, compared to 34.3% in the external VHA cohort (<i>N</i> = 397 150). The model's AUROC decreased from 0.74 in the internal test cohort to 0.70 in the full external cohort. Subgroup-level performance averaged 0.69 (SD = 0.01), showing minimal deviation from the external cohort overall. Retraining on VHA data improved AUROCs to 0.82. Clinical utility analysis showed systematic shifts in net-benefit across threshold probabilities.</p><p><strong>Discussion: </strong>While the POU model showed generalizability and fairness internally, external validation and retraining revealed performance and utility shifts across subgroups.</p><p><strong>Conclusion: </strong>Population-specific biases affect clinical utility-an often-overlooked dimension in fairness evaluation-a key need to ensure equitable benefits across diverse patient groups.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"8 5","pages":"ooaf115"},"PeriodicalIF":3.4,"publicationDate":"2025-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12483547/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145207911","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}