Iyad Majid, Vaibhav Mishra, Rohith Ravindranath, Sophia Y Wang
{"title":"Evaluating the Performance of Large Language Models for Named Entity Recognition in Ophthalmology Clinical Free-Text Notes.","authors":"Iyad Majid, Vaibhav Mishra, Rohith Ravindranath, Sophia Y Wang","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>This study compared large language models (LLMs) and Bidirectional Encoder Representations from Transformers (BERT) models in identifying medication names, routes, and frequencies from publicly available free-text ophthalmology progress notes of 480 patients. 5,520 lines of annotated text were divided into train (N=3,864), validation (N=1,104), and test sets (N=552). We evaluated ChatGPT-3.5, ChatGPT-4, PaLM 2, and Gemini to identify these medication entities. We fine-tuned BERT, BioBERT, ClinicalBERT, DistilBERT, and RoBERTa for the same task using the training set. On the test set, GPT-4 achieved the best performance (macro-averaged F1 0.962). Among the BERT models, BioBERT achieved the best performance (macro-averaged F1 0.875). Modern LLMs outperformed BERT models even in the highly domain-specific task of identifying ophthalmic medication information from progress notes, showcasing the potential of LLMs for medical named entity recognition to enhance patient care.</p>","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"778-787"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12099357/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144144562","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Hanna Kiani, Sohaib Hassan, Julian Z Genkins, Jasmine Bilir, Julia Kadie, Tran Le, Jo-Anne Suffoletto, Jonathan H Chen
{"title":"Improving Emergency Department Visit Risk Prediction: Exploring the Operational Utility of Applied Patient Portal Messages.","authors":"Hanna Kiani, Sohaib Hassan, Julian Z Genkins, Jasmine Bilir, Julia Kadie, Tran Le, Jo-Anne Suffoletto, Jonathan H Chen","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>Patient portal messages represent a unique source of clinical data due to how they represent the voice of the patient, provide a glimpse into care delivery between episodic synchronous appointments, and capture variations in patient behavior and health literacy. There is little understanding of how to best apply modern natural language processing (NLP) approaches, such as large, pre-trained language models (LLMs), to patient messages. In this study, we aim to explore different approaches in incorporating patient messages into an existing Emergency Departments (ED) visit risk prediction model currently deployed at Stanford Health Care. With the addition of patient message frequencies to the baseline we were able to achieve an improved AUC of .77 and a jump in the F1 score. In future work, we aim to build upon these findings and further test combination models to incorporate features around patient message content, in addition to message frequencies.</p>","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"610-619"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12099376/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144144673","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Varsha Varkhedi, Kenrick Cato, David Albers, Victoria L Tiase, Shalmali Joshi, Jennifer Thate, Kathryn Connell, William Hull, Amy Finnegan, Sarah C Rossetti
{"title":"Translating Nursing Data into Computational Metrics: An Evaluation Guideline for Inpatient Intravenous and Subcutaneous Insulin Management.","authors":"Varsha Varkhedi, Kenrick Cato, David Albers, Victoria L Tiase, Shalmali Joshi, Jennifer Thate, Kathryn Connell, William Hull, Amy Finnegan, Sarah C Rossetti","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>A challenge in utilizing electronic health record data for artificial intelligence models is contextualization, including understanding differences between missing data and missed care. Our team aims to develop knowledge graphs and computational models that account for these contexts, such as when data is missing (nurses being unable to document), but acceptable nursing care was delivered. We developed evaluation guidelines for intravenous and subcutaneous insulin management to establish a binary variable derived from EHR data representing minimally acceptable safe and quality nursing care for use in computational modeling. These guidelines were developed by our nurse informatics team based on best practices and validated by three nurse subject matter experts. The resulting evaluation guidelines are agnostic to institutional policies and focus on evaluating minimally acceptable safe and quality levels of care to inform inferences about missing data versus missed nursing care. Future work includes data-driven validations and expanding to other clinical scenarios.</p>","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"1295-1304"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12919408/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147273125","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Ziqing Ji, Patrick Wedgeworth, Kathleen Mertens, Sara L Jackson, Nkem O Akinsoto, Jared W Klein, Margaret L Isaac, Andrea L Hartzler
{"title":"EHR-Based Social Needs Screening and Referral in Primary Care: Clinician and Staff Perspectives on Practices, Barriers, and Benefits.","authors":"Ziqing Ji, Patrick Wedgeworth, Kathleen Mertens, Sara L Jackson, Nkem O Akinsoto, Jared W Klein, Margaret L Isaac, Andrea L Hartzler","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>Identifying social drivers of health (SDoH) can help healthcare professionals connect patients with community resources that impact health outcomes. However, screening and referral require time and effort, highlighting the potential for electronic health record (EHR) support to improve efficacy. We surveyed primary care clinicians and staff (n=122) about current SDoH screening and referral practices, as well as perceived barriers and benefits for standardizing those practices with EHR support. Although the majority currently screen and refer patients for SDoH at least occasionally, most respondents document in the EHR using unstructured formats and rate current practices as only moderately feasible and acceptable. While the majority feel competent with SDoH screening, key barriers included time constraints, lack of dedicated staff, and a desire for access to more community resources. Findings underscore the need for an EHR-based standardized screening and referral implementation with flexibility to adapt workflows in different clinic settings and clinical positions.</p>","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"536-545"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12919588/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147273114","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"From Food to Clinic: Mapping FoodOn to the UMLS to Enable Nutritional Decision Support.","authors":"Indra Neil Sarkar","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>Food and nutrition knowledge is recognized as a fundamental factor for the health and well-being of communities; however, its integration into biomedical and health knowledge systems is limited by the absence of standardized ontologies that encapsulate food-related concepts. This study mapped FoodOn, an open-source food ontology, to the Unified Medical Language System (UMLS) metathesaurus, a compendium of biomedical ontologies. As the first systematic mapping of a food ontology to the UMLS, the results of this study provide an ontological foundation for incorporating dietary data into clinical and public health workflows. The findings suggest that expanding the representation of food concepts in biomedical ontologies could enhance the potential to incorporate food and nutritional into clinical decision-making and research. Furthermore, this work lays the groundwork for integrating food-based therapies from traditional medicine systems (e.g., Ayurveda and Traditional Chinese Medicine) into contemporary clinical knowledge frameworks to support more holistic approaches to health care.</p>","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"1120-1129"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12919546/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147273113","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Healthy Lifestyles and Self-Improvement Videos on YouTube: A Thematic Analysis of Teen-Targeted Social Media Content.","authors":"Kyuha Jung, Tyler Kim, Yunan Chen","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>As teenagers increasingly turn to social media for health-related information, understanding the values of teen-targeted content has become important. Although videos on healthy lifestyles and self-improvement are gaining popularity on social media platforms like YouTube, little is known about how these videos benefit and engage with teenage viewers. To address this, we conducted a thematic analysis of 44 YouTube videos and 66,901 comments. We found that these videos provide various advice on teenagers' common challenges, use engaging narratives for authenticity, and foster teen-centered communities through comments. However, a few videos also gave misleading advice to adolescents that can be potentially harmful. Based on our findings, we discuss design implications for creating relatable and intriguing social media content for adolescents. Additionally, we suggest ways for social media platforms to promote healthier and safer experiences for teenagers.</p>","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"556-565"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12919577/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147273081","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Sachleen K Tuteja, Elise L Boventer, Abdulaziz Alkattan, Noemie Elhadad, Sarah C Rossetti
{"title":"Shifting Information Needs in Clinical Practice: The Evolving Role of Generative AI in Addressing Clinician Demands for Context-Specific Knowledge.","authors":"Sachleen K Tuteja, Elise L Boventer, Abdulaziz Alkattan, Noemie Elhadad, Sarah C Rossetti","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>This study explores clinicians' evolving information needs and evaluates the potential of Generative Artificial Intelligence (Gen AI) to address these gaps by reassessing and extending the Currie et al. (2003) taxonomy. Despite advancements in electronic health records (EHRs), unresolved information needs persist, impacting clinical efficiency and patient care. A cross-sectional survey conducted at Columbia University Irving Medical Center (CUIMC) analyzed clinician-generated Gen AI prompts, comparing them against the 2003 taxonomy. Findings reveal that while 80% of prompts align with existing categories, 20% represent emerging needs, including AI-driven workflow optimization and fairness-related inquiries. These findings highlight the necessity of adapting clinical decision support frameworks to integrate AI-driven solutions, ensuring that modern tools meet evolving clinician needs. By formally extending the Currie et al. taxonomy, this study provides a foundational framework for leveraging Gen AI to bridge long-standing information gaps and enhance patient outcomes in an increasingly complex healthcare environment.</p>","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"1277-1284"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12919614/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147273076","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Sonish Sivarajkumar, Kimia Ameri, Chuqin Li, Yanshan Wang, Min Jiang
{"title":"Automating Adjudication of Cardiovascular Events Using Large Language Models.","authors":"Sonish Sivarajkumar, Kimia Ameri, Chuqin Li, Yanshan Wang, Min Jiang","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>Cardiovascular event adjudication is essential in clinical trials but relies on manual chart review that is slow, variable, and expensive. We present a two-stage framework that automates adjudication of cardiovascular deaths using large language models (LLMs). First, a few-shot LLM extracts structured evidence (event, span, negation, date) from unstructured clinical documents. Second, a Tree-of-Thoughts adjudicator aligns its reasoning with clinical endpoint committee (CEC) guidelines to classify deaths as cardiovascular or non-cardiovascular and produce an auditable rationale. On Lilly clinical-trial data, extraction achieved precision 0.96, recall 0.71 (F1 0.82), and adjudication attained 0.68 accuracy (GPT-4 ToT), outperforming a summarizer-plus-adjudicator baseline. We introduce CLEART, a rubric-based automated score that quantifies rationale quality across clarity, consistency, detail, guideline adherence, relevance, and timeline accuracy (overall 0.67), highlighting temporal reasoning and relevance as key areas for improvement. This approach can reduce adjudication time and variability while increasing transparency.</p>","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"1219-1228"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12919421/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147273197","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Improving electronic health record processing of large language models via retrieval-augmented generation: A case study on dietary supplements.","authors":"Zaifu Zhan, Shuang Zhou, Jiawen Deng, Rui Zhang","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>Large language models (LLMs) excel in natural language processing (NLP) but struggle with domain-specific complexities in electronic health records (EHRs). We demonstrate that retrieval-augmented generation (RAG) enhances LLMs for dietary supplement (DS) information extraction. By testing models like Llama-3 with diverse retrievers on tasks including entity recognition and usage classification, task-aligned retrieval outperforms reliance on model size or specialization. Smaller general models paired with optimized retrievers match or exceed specialized counterparts-structured retrieval aids complex tasks (e.g., triple extraction), while semantic retrieval improves classification. Results challenge assumptions that larger or domain-specific models are superior, emphasizing dynamic knowledge integration over brute-force scaling. This approach offers practical strategies for clinical NLP, enabling efficient EHR analysis without massive resources. Prioritizing retrieval strategies over model size advances tools for evidence-based healthcare, highlighting adaptability and cost-effectiveness in real-world medical applications.</p>","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"1511-1518"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12919604/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147273093","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Youngjin Lee, Min-Jeoung Kang, Veysel K Baris, Graham Lowenthal, Sarah C Rossetti, Kenrick D Cato, Rachel Y Lee, Janna Kramer, Richard Huffam, Patricia C Dykes
{"title":"Exploring the Implementation Experience and Use of CONCERN Early Warning System in a Rural Community Hospital: A Mixed Method Convergent Approach.","authors":"Youngjin Lee, Min-Jeoung Kang, Veysel K Baris, Graham Lowenthal, Sarah C Rossetti, Kenrick D Cato, Rachel Y Lee, Janna Kramer, Richard Huffam, Patricia C Dykes","doi":"","DOIUrl":"","url":null,"abstract":"<p><p>The Communicating Narrative Concerns Entered by RNs Early Warning System (CONCERN EWS) is a machine-learning predictive model that analyzes nursing documentation patterns to detect early signs of patient deterioration, with proven effectiveness in reducing the risk of in-hospital mortality and length of stay. This study extends the evaluation of CONCERN EWS beyond acute care settings to a rural community hospital, assessing user experience, system utilization, and accuracy in identifying patient deterioration. The study examined accuracy in a rural setting using a mixed-methods approach-qualitative interviews, quantitative data analysis, and clinical record reviews. Findingssuggestthat CONCERNEWS enhancesearly recognition of clinical deterioration andisusable in the context of busy acute care nursing workflows. These results support its adaptability to facilitate strengthening nursing surveillance, clinical decision-making, and patient safety in rural healthcare settings as well.</p>","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"724-733"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12919451/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147273107","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}