Journal of the American Medical Informatics Association最新文献

{"title":"Robust privacy amidst innovation with large language models through a critical assessment of the risks.","authors":"Yao-Shun Chuang, Atiquer Rahman Sarkar, Yu-Chun Hsu, Noman Mohammed, Xiaoqian Jiang","doi":"10.1093/jamia/ocaf037","DOIUrl":"https://doi.org/10.1093/jamia/ocaf037","url":null,"abstract":"<p><strong>Objective: </strong>This study evaluates the integration of electronic health records (EHRs) and natural language processing (NLP) with large language models (LLMs) to enhance healthcare data management and patient care, focusing on using advanced language models to create secure, Health Insurance Portability and Accountability Act-compliant synthetic patient notes for global biomedical research.</p><p><strong>Materials and methods: </strong>The study used de-identified and re-identified versions of the MIMIC III dataset with GPT-3.5, GPT-4, and Mistral 7B to generate synthetic clinical notes. Text generation employed templates and keyword extraction for contextually relevant notes, with One-shot generation for comparison. Privacy was assessed by analyzing protected health information (PHI) occurrence and co-occurrence, while utility was evaluated by training an ICD-9 coder using synthetic notes. Text quality was measured using ROUGE (Recall-Oriented Understudy for Gisting Evaluation) and cosine similarity metrics to compare synthetic notes with source notes for semantic similarity.</p><p><strong>Results: </strong>The analysis of PHI occurrence and text utility via the ICD-9 coding task showed that the keyword-based method had low risk and good performance. One-shot generation exhibited the highest PHI exposure and PHI co-occurrence, particularly in geographic location and date categories. The Normalized One-shot method achieved the highest classification accuracy. Re-identified data consistently outperformed de-identified data.</p><p><strong>Discussion: </strong>Privacy analysis revealed a critical balance between data utility and privacy protection, influencing future data use and sharing.</p><p><strong>Conclusion: </strong>This study shows that keyword-based methods can create synthetic clinical notes that protect privacy while retaining data usability, potentially improving clinical data sharing. The use of dummy PHIs to counter privacy attacks may offer better utility and privacy than traditional de-identification.</p>","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":4.7,"publicationDate":"2025-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143671455","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Utilizing large language models for detecting hospital-acquired conditions: an empirical study on pulmonary embolism.","authors":"Cheligeer Cheligeer, Danielle A Southern, Jun Yan, Guosong Wu, Jie Pan, Seungwon Lee, Elliot A Martin, Hamed Jafarpour, Cathy A Eastwood, Yong Zeng, Hude Quan","doi":"10.1093/jamia/ocaf048","DOIUrl":"https://doi.org/10.1093/jamia/ocaf048","url":null,"abstract":"&lt;p&gt;&lt;strong&gt;Objectives: &lt;/strong&gt;Adverse event detection from Electronic Medical Records (EMRs) is challenging due to the low incidence of the event, variability in clinical documentation, and the complexity of data formats. Pulmonary embolism as an adverse event (PEAE) is particularly difficult to identify using existing approaches. This study aims to develop and evaluate a Large Language Model (LLM)-based framework for detecting PEAE from unstructured narrative data in EMRs.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Materials and methods: &lt;/strong&gt;We conducted a chart review of adult patients (aged 18-100) admitted to tertiary-care hospitals in Calgary, Alberta, Canada, between 2017-2022. We developed an LLM-based detection framework consisting of three modules: evidence extraction (implementing both keyword-based and semantic similarity-based filtering methods), discharge information extraction (focusing on six key clinical sections), and PEAE detection. Four open-source LLMs (Llama3, Mistral-7B, Gemma, and Phi-3) were evaluated using positive predictive value, sensitivity, specificity, and F1-score. Model performance for population-level surveillance was assessed at yearly, quarterly, and monthly granularities.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Results: &lt;/strong&gt;The chart review included 10 066 patients, with 40 cases of PEAE identified (0.4% prevalence). All four LLMs demonstrated high sensitivity (87.5-100%) and specificity (94.9-98.9%) across different experimental conditions. Gemma achieved the highest F1-score (28.11%) using keyword-based retrieval with discharge summary inclusion, along with 98.4% specificity, 87.5% sensitivity, and 99.95% negative predictive value. Keyword-based filtering reduced the median chunks per patient from 789 to 310, while semantic filtering further reduced this to 9 chunks. Including discharge summaries improved performance metrics across most models. For population-level surveillance, all models showed strong correlation with actual PEAE trends at yearly granularity (r=0.92-0.99), with Llama3 achieving the highest correlation (0.988).&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Discussion: &lt;/strong&gt;The results of our method for PEAE detection using EMR notes demonstrate high sensitivity and specificity across all four tested LLMs, indicating strong performance in distinguishing PEAE from non-PEAE cases. However, the low incidence rate of PEAE contributed to a lower PPV. The keyword-based chunking approach consistently outperformed semantic similarity-based methods, achieving higher F1 scores and PPV, underscoring the importance of domain knowledge in text segmentation. Including discharge summaries further enhanced performance metrics. Our population-based analysis revealed better performance for yearly trends compared to monthly granularity, suggesting the framework's utility for long-term surveillance despite dataset imbalance. Error analysis identified contextual misinterpretation, terminology confusion, and preprocessing limitations as key challenges for future improvement.&lt;/p&gt;&lt;p&gt;&lt;","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":4.7,"publicationDate":"2025-03-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143659571","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Patient and clinician acceptability of automated extraction of social drivers of health from clinical notes in primary care.","authors":"Serena Jinchen Xie, Carolin Spice, Patrick Wedgeworth, Raina Langevin, Kevin Lybarger, Angad Preet Singh, Brian R Wood, Jared W Klein, Gary Hsieh, Herbert C Duber, Andrea L Hartzler","doi":"10.1093/jamia/ocaf046","DOIUrl":"https://doi.org/10.1093/jamia/ocaf046","url":null,"abstract":"<p><strong>Objective: </strong>Artificial Intelligence (AI)-based approaches for extracting Social Drivers of Health (SDoH) from clinical notes offer healthcare systems an efficient way to identify patients' social needs, yet we know little about the acceptability of this approach to patients and clinicians. We investigated patient and clinician acceptability through interviews.</p><p><strong>Materials and methods: </strong>We interviewed primary care patients experiencing social needs (n = 19) and clinicians (n = 14) about their acceptability of \"SDoH autosuggest,\" an AI-based approach for extracting SDoH from clinical notes. We presented storyboards depicting the approach and asked participants to rate their acceptability and discuss their rationale.</p><p><strong>Results: </strong>Participants rated SDoH autosuggest moderately acceptable (mean = 3.9/5 patients; mean = 3.6/5 clinicians). Patients' ratings varied across domains, with substance use rated most and employment rated least acceptable. Both groups raised concern about information integrity, actionability, impact on clinical interactions and relationships, and privacy. In addition, patients raised concern about transparency, autonomy, and potential harm, whereas clinicians raised concern about usability.</p><p><strong>Discussion: </strong>Despite reporting moderate acceptability of the envisioned approach, patients and clinicians expressed multiple concerns about AI systems that extract SDoH. Participants emphasized the need for high-quality data, non-intrusive presentation methods, and clear communication strategies regarding sensitive social needs. Findings underscore the importance of engaging patients and clinicians to mitigate unintended consequences when integrating AI approaches into care.</p><p><strong>Conclusion: </strong>Although AI approaches like SDoH autosuggest hold promise for efficiently identifying SDoH from clinical notes, they must also account for concerns of patients and clinicians to ensure these systems are acceptable and do not undermine trust.</p>","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":4.7,"publicationDate":"2025-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143626628","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Unmet social needs and diverticulitis: a phenotyping algorithm and cross-sectional analysis.","authors":"Thomas E Ueland, Samuel A Younan, Parker T Evans, Jessica Sims, Megan M Shroder, Alexander T Hawkins, Richard Peek, Xinnan Niu, Lisa Bastarache, Jamie R Robinson","doi":"10.1093/jamia/ocae238","DOIUrl":"https://doi.org/10.1093/jamia/ocae238","url":null,"abstract":"<p><strong>Objective: </strong>To validate a phenotyping algorithm for gradations of diverticular disease severity and investigate relationships between unmet social needs and disease severity.</p><p><strong>Materials and methods: </strong>An algorithm was designed in the All of Us Research Program to identify diverticulosis, mild diverticulitis, and operative or recurrent diverticulitis requiring multiple inpatient admissions. This was validated in an independent institution and applied to a cohort in the All of Us Research Program. Distributions of individual-level social barriers were compared across quintiles of an area-level index through fold enrichment of the barrier in the fifth (most deprived) quintile relative to the first (least deprived) quintile. Social needs of food insecurity, housing instability, and care access were included in logistic regression to assess association with disease severity.</p><p><strong>Results: </strong>Across disease severity groups, the phenotyping algorithm had positive predictive values ranging from 0.87 to 0.97 and negative predictive values ranging from 0.97 to 0.99. Unmet social needs were variably distributed when comparing the most to the least deprived quintile of the area-level deprivation index (fold enrichment ranging from 0.53 to 15). Relative to a reference of diverticulosis, an unmet social need was associated with greater odds of operative or recurrent inpatient diverticulitis (OR [95% CI] 1.61 [1.19-2.17]).</p><p><strong>Discussion: </strong>Understanding the landscape of social barriers in disease-specific cohorts may facilitate a targeted approach when addressing these needs in clinical settings.</p><p><strong>Conclusion: </strong>Using a validated phenotyping algorithm for diverticular disease severity, unmet social needs were found to be associated with greater severity of diverticulitis presentation.</p>","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":4.7,"publicationDate":"2025-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143626629","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A call for the informatics community to define priority practice and research areas at the intersection of climate and health: report from 2023 mini-summit.","authors":"Titus Schleyer, Manijeh Berenji, Monica Deck, Hana Chung, Joshua Choi, Theresa A Cullen, Timothy Burdick, Amanda Zaleski, Kelly Jean Thomas Craig, Oluseyi Fayanju, Muhammad Muinul Islam","doi":"10.1093/jamia/ocae292","DOIUrl":"https://doi.org/10.1093/jamia/ocae292","url":null,"abstract":"<p><strong>Objective: </strong>Although biomedical informatics has multiple roles to play in addressing the climate crisis, collaborative action and research agendas have yet to be developed. As a first step, AMIA's new Climate, Health, and Informatics Working Group held a mini-summit entitled Climate and health: How can informatics help? during the AMIA 2023 Fall Symposium to define an initial set of areas of interest and begin mobilizing informaticians to confront the urgent challenges of climate change.</p><p><strong>Materials and methods: </strong>The AMIA Climate, Health, and Informatics Working Group (at the time, an AMIA Discussion Forum), the International Medical Informatics Association (IMIA), the International Academy of Health Sciences Informatics (IAHSI), and the Regenstrief Institute hosted a mini-summit entitled Climate and health: How can informatics help? on November 11, 2023, during the AMIA 2023 Annual Symposium (New Orleans, LA, USA). Using an affinity diagramming approach, the mini-summit organizers posed 2 questions to ∼50 attendees (40 in-person, 10 virtual).</p><p><strong>Results: </strong>Participants expressed a broad array of viewpoints on actions that can be undertaken now and areas needing research to support future actions. Areas of current action ranged from enhanced education to expanded telemedicine to assessment of community vulnerability. Areas of research ranged from emergency preparedness to climate-specific clinical coding to risk prediction models.</p><p><strong>Discussion: </strong>The mini-summit was intended as a first step in helping the informatics community at large set application and research priorities for climate, health, and informatics.</p><p><strong>Conclusion: </strong>The working group will use these perspectives as it seeks further input, and begins to establish priorities for climate-related biomedical informatics actions and research.</p>","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":4.7,"publicationDate":"2025-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143626625","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Optimizing the efficiency and effectiveness of data quality assurance in a multicenter clinical dataset.","authors":"Anne Fu, Trong Shen, Surain B Roberts, Weihan Liu, Shruthi Vaidyanathan, Kayley-Jasmin Marchena-Romero, Yuen Yu Phyllis Lam, Kieran Shah, Denise Y F Mak, Fahad Razak, Amol A Verma","doi":"10.1093/jamia/ocaf042","DOIUrl":"https://doi.org/10.1093/jamia/ocaf042","url":null,"abstract":"<p><strong>Objectives: </strong>Electronic health records (EHRs) data are increasingly used for research and analysis, but there is little empirical evidence to inform how automated and manual assessments can be combined to efficiently assess data quality in large EHR repositories.</p><p><strong>Materials and methods: </strong>The GEMINI database collected data from 462 226 patient admissions across 32 hospitals from 2021 to 2023. We report data quality issues identified through semi-automated and manual data quality assessments completed during the data collection phase. We conducted a simulation experiment to evaluate the relationship between the number of records reviewed manually, the detection of true data errors (true positives) and the number of manual chart abstraction errors (false positives) that required unnecessary investigation.</p><p><strong>Results: </strong>The semi-automated data quality assessments identified 79 data quality issues requiring correction, of which 14 had a large impact, affecting at least 50% of records in the data. After resolving issues identified through semi-automated assessments, manual validation of 2676 patient encounters at 19 hospitals identified 4 new meaningful data errors (3 in transfusion data and 1 in physician identifiers), distributed across 4 hospitals. There were 365 manual chart abstraction errors, which required investigation by data analysts to identify as \"false positives.\" These errors increased linearly with the number of charts reviewed manually. Simulation results demonstrate that all 3 transfusion data errors were identified with 95% sensitivity after manual review of 5 records, whereas 18 records were needed for the physician's table.</p><p><strong>Discussion and conclusion: </strong>The GEMINI approach represents a scalable framework for data quality assessment and improvement in multisite EHR research databases. Manual data review is important but can be minimized to optimize the trade-off between true and false identification of data quality errors.</p>","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":4.7,"publicationDate":"2025-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143626627","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Emerging algorithmic bias: fairness drift as the next dimension of model maintenance and sustainability.","authors":"Sharon E Davis, Chad Dorn, Daniel J Park, Michael E Matheny","doi":"10.1093/jamia/ocaf039","DOIUrl":"https://doi.org/10.1093/jamia/ocaf039","url":null,"abstract":"<p><strong>Objectives: </strong>While performance drift of clinical prediction models is well-documented, the potential for algorithmic biases to emerge post-deployment has had limited characterization. A better understanding of how temporal model performance may shift across subpopulations is required to incorporate fairness drift into model maintenance strategies.</p><p><strong>Materials and methods: </strong>We explore fairness drift in a national population over 11 years, with and without model maintenance aimed at sustaining population-level performance. We trained random forest models predicting 30-day post-surgical readmission, mortality, and pneumonia using 2013 data from US Department of Veterans Affairs facilities. We evaluated performance quarterly from 2014 to 2023 by self-reported race and sex. We estimated discrimination, calibration, and accuracy, and operationalized fairness using metric parity measured as the gap between disadvantaged and advantaged groups.</p><p><strong>Results: </strong>Our cohort included 1 739 666 surgical cases. We observed fairness drift in both the original and temporally updated models. Model updating had a larger impact on overall performance than fairness gaps. During periods of stable fairness, updating models at the population level increased, decreased, or did not impact fairness gaps. During periods of fairness drift, updating models restored fairness in some cases and exacerbated fairness gaps in others.</p><p><strong>Discussion: </strong>This exploratory study highlights that algorithmic fairness cannot be assured through one-time assessments during model development. Temporal changes in fairness may take multiple forms and interact with model updating strategies in unanticipated ways.</p><p><strong>Conclusion: </strong>Equitable and sustainable clinical artificial intelligence deployments will require novel methods to monitor algorithmic fairness, detect emerging bias, and adopt model updates that promote fairness.</p>","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":4.7,"publicationDate":"2025-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143626626","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Letter to the editors in response to \"Leveraging artificial intelligence to summarize abstracts in lay language for increasing research accessibility and transparency\".","authors":"Ethan Layne, Francesco Cei, Giovanni E Cacciamani","doi":"10.1093/jamia/ocaf024","DOIUrl":"https://doi.org/10.1093/jamia/ocaf024","url":null,"abstract":"","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":4.7,"publicationDate":"2025-03-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143617684","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Associations of perceived discrimination with health outcomes and health disparities in the All of Us cohort.","authors":"Vincent Lam, Sonali Gupta, I King Jordan, Leonardo Mariño-Ramírez","doi":"10.1093/jamia/ocaf040","DOIUrl":"10.1093/jamia/ocaf040","url":null,"abstract":"<p><strong>Objectives: </strong>The goal of this study was to investigate the association of perceived discrimination with health outcomes and disparities.</p><p><strong>Materials and methods: </strong>The study cohort consists of 60 180 participants from the 4 largest self-identified race and ethnicity (SIRE) groups in the All of Us Research Program participant body: Asian (1291), Black (4726), Hispanic (5336), and White (48 827). A perceived discrimination index (PDI) was derived from participant responses to the \"Social Determinants of Health\" survey, and the All of Us Researcher Workbench was used to analyze associations and mediation effects of PDI and SIRE with 1755 diseases.</p><p><strong>Results: </strong>The Black SIRE group has the greatest median PDI, followed by the Asian, Hispanic, and White groups. The Black SIRE group shows the greatest number of diseases with elevated risk relative to the White reference group, followed by the Hispanic and Asian groups. Perceived discrimination index was found to be positively and significantly associated with 489 out of 1755 (27.86%) diseases. \"Mental Disorders\" is the disease category with the greatest proportion of diseases positively and significantly associated with PDI: 59 out of 72 (81.94%) diseases. Mediation analysis showed that PDI mediates 69 out of 351 (19.66%) Black-White disease disparities.</p><p><strong>Discussion: </strong>Perceived discrimination is significantly associated with risk for numerous diseases and mediates Black-White disease disparities in the All of Us participant cohort.</p><p><strong>Conclusion: </strong>This work highlights the role of discrimination as an important social determinant of health and provides a means by which it can be quantified and modeled on the All of Us platform.</p>","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":4.7,"publicationDate":"2025-03-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143617674","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Application of unified health large language model evaluation framework to In-Basket message replies: bridging qualitative and quantitative assessments.","authors":"Chuan Hong, Anand Chowdhury, Anthony D Sorrentino, Haoyuan Wang, Monica Agrawal, Armando Bedoya, Sophia Bessias, Nicoleta J Economou-Zavlanos, Ian Wong, Christian Pean, Fan Li, Kathryn I Pollak, Eric G Poon, Michael J Pencina","doi":"10.1093/jamia/ocaf023","DOIUrl":"https://doi.org/10.1093/jamia/ocaf023","url":null,"abstract":"<p><strong>Objectives: </strong>Large language models (LLMs) are increasingly utilized in healthcare, transforming medical practice through advanced language processing capabilities. However, the evaluation of LLMs predominantly relies on human qualitative assessment, which is time-consuming, resource-intensive, and may be subject to variability and bias. There is a pressing need for quantitative metrics to enable scalable, objective, and efficient evaluation.</p><p><strong>Materials and methods: </strong>We propose a unified evaluation framework that bridges qualitative and quantitative methods to assess LLM performance in healthcare settings. This framework maps evaluation aspects-such as linguistic quality, efficiency, content integrity, trustworthiness, and usefulness-to both qualitative assessments and quantitative metrics. We apply our approach to empirically evaluate the Epic In-Basket feature, which uses LLM to generate patient message replies.</p><p><strong>Results: </strong>The empirical evaluation demonstrates that while Artificial Intelligence (AI)-generated replies exhibit high fluency, clarity, and minimal toxicity, they face challenges with coherence and completeness. Clinicians' manual decision to use AI-generated drafts correlates strongly with quantitative metrics, suggesting that quantitative metrics have the potential to reduce human effort in the evaluation process and make it more scalable.</p><p><strong>Discussion: </strong>Our study highlights the potential of a unified evaluation framework that integrates qualitative and quantitative methods, enabling scalable and systematic assessments of LLMs in healthcare. Automated metrics streamline evaluation and monitoring processes, but their effective use depends on alignment with human judgment, particularly for aspects requiring contextual interpretation. As LLM applications expand, refining evaluation strategies and fostering interdisciplinary collaboration will be critical to maintaining high standards of accuracy, ethics, and regulatory compliance.</p><p><strong>Conclusion: </strong>Our unified evaluation framework bridges the gap between qualitative human assessments and automated quantitative metrics, enhancing the reliability and scalability of LLM evaluations in healthcare. While automated quantitative evaluations are not ready to fully replace qualitative human evaluations, they can be used to enhance the process and, with relevant benchmarks derived from the unified framework proposed here, they can be applied to LLM monitoring and evaluation of updated versions of the original technology evaluated using qualitative human standards.</p>","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":4.7,"publicationDate":"2025-03-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143587876","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}