AMIA ... Annual Symposium proceedings. AMIA Symposium最新文献

Large Language Models Struggle in Token-Level Clinical Named Entity Recognition. 大型语言模型在符号级临床命名实体识别中的挣扎。

AMIA ... Annual Symposium proceedings. AMIA Symposium Pub Date : 2025-05-22 eCollection Date: 2024-01-01

Qiuhao Lu, Rui Li, Andrew Wen, Jinlian Wang, Liwei Wang, Hongfang Liu

{"title":"Large Language Models Struggle in Token-Level Clinical Named Entity Recognition.","authors":"Qiuhao Lu, Rui Li, Andrew Wen, Jinlian Wang, Liwei Wang, Hongfang Liu","doi":"","DOIUrl":"","url":null,"abstract":"Large Language Models (LLMs) have revolutionized various sectors, including healthcare where they are employed in diverse applications. Their utility is particularly significant in the context of rare diseases, where data scarcity, complexity, and specificity pose considerable challenges. In the clinical domain, Named Entity Recognition (NER) stands out as an essential task and it plays a crucial role in extracting relevant information from clinical texts. Despite the promise of LLMs, current research mostly concentrates on document-level NER, identifying entities in a more general context across entire documents, without extracting their precise location. Additionally, efforts have been directed towards adapting ChatGPTfor token-level NER. However, there is a significant research gap when it comes to employing token-level NER for clinical texts, especially with the use of local open-source LLMs. This study aims to bridge this gap by investigating the effectiveness of both proprietary and local LLMs in token-level clinical NER. Essentially, we delve into the capabilities of these models through a series of experiments involving zero-shot prompting, few-shot prompting, retrieval-augmented generation (RAG), and instruction-fine-tuning. Our exploration reveals the inherent challenges LLMs face in token-level NER, particularly in the context of rare diseases, and suggests possible improvements for their application in healthcare. This research contributes to narrowing a significant gap in healthcare informatics and offers insights that could lead to a more refined application of LLMs in the healthcare sector.","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"748-757"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12099373/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144144361","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Development of a Human Evaluation Framework and Correlation with Automated Metrics for Natural Language Generation of Medical Diagnoses. 医学诊断自然语言生成的人类评估框架及其与自动度量的关联。

AMIA ... Annual Symposium proceedings. AMIA Symposium Pub Date : 2025-05-22 eCollection Date: 2024-01-01

Emma Croxford, Yanjun Gao, Brian Patterson, Daniel To, Samuel Tesch, Dmitriy Dligach, Anoop Mayampurath, Matthew M Churpek, Majid Afshar

{"title":"Development of a Human Evaluation Framework and Correlation with Automated Metrics for Natural Language Generation of Medical Diagnoses.","authors":"Emma Croxford, Yanjun Gao, Brian Patterson, Daniel To, Samuel Tesch, Dmitriy Dligach, Anoop Mayampurath, Matthew M Churpek, Majid Afshar","doi":"","DOIUrl":"","url":null,"abstract":"In the evolving landscape of clinical Natural Language Generation (NLG), assessing abstractive text quality remains challenging, as existing methods often overlook generative task complexities. This work aimed to examine the current state of automated evaluation metrics in NLG in healthcare. To have a robust and well-validated baseline with which to examine the alignment of these metrics, we created a comprehensive human evaluation framework. Employing ChatGPT-3.5-turbo generative output, we correlated human judgments with each metric. None of the metrics demonstrated high alignment; however, the SapBERT score-a Unified Medical Language System (UMLS)- showed the best results. This underscores the importance of incorporating domain-specific knowledge into evaluation efforts. Our work reveals the deficiency in quality evaluations for generated text and introduces our comprehensive human evaluation framework as a baseline. Future efforts should prioritize integrating medical knowledge databases to enhance the alignment of automated metrics, particularly focusing on refining the SapBERT score for improved assessments.","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"309-318"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12099413/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144144496","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Lessons Learned from OpenEMR Implementation in Graduate Health Informatics Curriculum. 研究生健康信息学课程实施openenemr的经验教训。

AMIA ... Annual Symposium proceedings. AMIA Symposium Pub Date : 2025-05-22 eCollection Date: 2024-01-01

Keerthika Sunchu, Megha M Moncy, Saptarshi Purkayastha, Cathy R Fulton

{"title":"Lessons Learned from OpenEMR Implementation in Graduate Health Informatics Curriculum.","authors":"Keerthika Sunchu, Megha M Moncy, Saptarshi Purkayastha, Cathy R Fulton","doi":"","DOIUrl":"","url":null,"abstract":"This study examines the integration of OpenEMR, a Meaningful Use-certified open-source electronic health record (EHR) system, into a Health Informatics curriculum. The primary objective was to address the disparity between theoretical knowledge and practical application in health informatics education. The implementation process revealed several significant challenges, including unintended system modifications that compromised functionality, data entry errors that impacted usability, and technical issues that impeded accessibility. To mitigate these challenges, a series of interventions were implemented. These included backend modifications to enhance data entry accuracy, usability improvements such as limiting open tabs to facilitate navigation, and the implementation ofproactive measures to expedite the resolution of technical issues. The experiences gained from this integration process highlight three critical aspects of health informatics education: the significance of practical proficiency in EHR systems, the necessity for user-centric interface design, and the importance of adaptability and problem-solving skills. The study proposes several future directions for research and practice. These include fostering global collaboration, developing standardized curricula for EHR education, and establishing robust mechanisms for continuous assessment and improvement. The findings underscore the pivotal role of integrating hands-on EHR experience into health informatics education, emphasizing its potential to equip students with the essential competencies required to navigate the complex and dynamic healthcare landscape.","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"1079-1088"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12099383/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144144577","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Enhancement of Fairness in AI for Chest X-ray Classification. 增强胸部x线分类人工智能公平性

AMIA ... Annual Symposium proceedings. AMIA Symposium Pub Date : 2025-05-22 eCollection Date: 2024-01-01

Nicholas J Jackson, Chao Yan, Bradley A Malin

引用次数: 0

Factors Driving Patient Decisions to Access Electronic Health Records via a Breast Cancer Online Decision Aid linked to the Patient Portal. 通过与患者门户网站链接的乳腺癌在线决策辅助，驱动患者决定访问电子健康记录的因素。

AMIA ... Annual Symposium proceedings. AMIA Symposium Pub Date : 2025-05-22 eCollection Date: 2024-01-01

Anna Vaynrub, Subiksha Umakanth, Harry West, Alissa Michel, Jill Dimond, Stephan Constante, Katherine D Crew, Rita Kukafka

{"title":"Factors Driving Patient Decisions to Access Electronic Health Records via a Breast Cancer Online Decision Aid linked to the Patient Portal.","authors":"Anna Vaynrub, Subiksha Umakanth, Harry West, Alissa Michel, Jill Dimond, Stephan Constante, Katherine D Crew, Rita Kukafka","doi":"","DOIUrl":"","url":null,"abstract":"A critical strategy in limiting breast cancer (BC) mortality is the early identification of high-risk patients and implementation of risk-reducing measures. RealRisks, an online decision aid constructed by our team to provide education on BC risk and personalized risk assessment, allows users the option to connect to their electronic health record (EHR) to extract requisite data to calculate BC risk via Fast Healthcare Interoperability Resources (FHIR). Using data from RealRisks user profiles, baseline questionnaires, and interview transcripts, we sought to understand the differences between the groups of patients who opted to download their data via the EHR vs. those who did not. A higher percentage of those who downloaded data (53.8% vs. 42.3%) identified as Hispanic/Latino compared to those who did not download. Thematic analysis suggested that while data security and privacy concerns may lead to hesitation, it is perhaps technological barriers that most significantly limit EHR download.","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"1159-1168"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12099358/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144144644","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Acceptability of pictographs as a novel patient identifier to improve patient safety in the neonatal intensive care unit. 可接受的象形文字作为一种新的患者标识符，以提高新生儿重症监护病房的患者安全。

AMIA ... Annual Symposium proceedings. AMIA Symposium Pub Date : 2025-05-22 eCollection Date: 2024-01-01

Hojjat Salmasian, Carmina Erdei, Joanne R Applebaum, Danielle Sharon, Katie Hannon, Deborah Cuddyer, Mary Sawyer, Tina Steele, Yvonne Sheldon, I-Fong S Lehman, Joseph E Schwartz, Allen Chen, Jason Adelman

{"title":"Acceptability of pictographs as a novel patient identifier to improve patient safety in the neonatal intensive care unit.","authors":"Hojjat Salmasian, Carmina Erdei, Joanne R Applebaum, Danielle Sharon, Katie Hannon, Deborah Cuddyer, Mary Sawyer, Tina Steele, Yvonne Sheldon, I-Fong S Lehman, Joseph E Schwartz, Allen Chen, Jason Adelman","doi":"","DOIUrl":"","url":null,"abstract":"As part of a randomized controlled trial on the use of pictographs (images used in lieu of a patient photo) embedded in the electronic health record to reduce wrong-patient errors in the neonatal intensive care unit (NICU), we conducted a series of surveys of parents, providers and nurses in the NICU. Data from survey responses were thematically analyzed and categorized. We found that in all groups, there was very high awareness of the intended purpose of the pictographs; however, the perception of providers and nurses about the effectiveness of pictographs was not as strong. While several providers and nurses acknowledged that pictographs can or have helped them avoid wrong-patient errors when caring for multiple birth infants (such as twins), many nurses believed that their current practice of the use of two patient identifiers was sufficient, and pictographs were not useful. Parents reported that pictographs improved their experience of care.","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"980-986"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12099398/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144144692","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Analyzing Dementia Caregivers' Experiences on Twitter: A Term-Weighted Topic Modeling Approach. 分析痴呆症护理者在Twitter上的经历：一种术语加权主题建模方法。

AMIA ... Annual Symposium proceedings. AMIA Symposium Pub Date : 2025-05-22 eCollection Date: 2024-01-01

Yanbo Feng, Bojian Hou, Ari Klein, Karen O'Connor, Jiong Chen, Andrées Mondragóon, Shu Yang, Graciela Gonzalez-Hernandez, Li Shen

{"title":"Analyzing Dementia Caregivers' Experiences on Twitter: A Term-Weighted Topic Modeling Approach.","authors":"Yanbo Feng, Bojian Hou, Ari Klein, Karen O'Connor, Jiong Chen, Andrées Mondragóon, Shu Yang, Graciela Gonzalez-Hernandez, Li Shen","doi":"","DOIUrl":"","url":null,"abstract":"Dementia profoundly impacts patients and their families, making it essential to understand the experiences and concerns offamily caregivers for enhanced support and care. This study introduces a novel approach to analyzing tweets from individuals whose family members suffer from dementia. We preprocessed our collected Twitter (now X) data using advanced natural language processing techniques and enhanced conventional topic model-Gibbs Sampling Dirichlet Multinomial Mixture Model (GSDMM)-with term-weighting strategies to improve topic clarity. This enhanced approach enabled the identification of key topics among dementia-affected families, offering semantically rich and contextually coherent topics, demonstrating that our method outperforms the state-of-the-art BERTopic model in clarity and consistency. Leveraging ChatGPT 4 alongside two human experts, we uncovered the multifaceted challenges faced by family caregivers. This work aims to provide healthcare professionals, researchers, and support organizations with a valuable tool to better understand and address the needs offamily caregivers.","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"407-416"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12099380/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144144696","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Narrative Feature or Structured Feature? A Study of Large Language Models to Identify Cancer Patients at Risk of Heart Failure. 叙事特征还是结构特征？大型语言模型识别心脏衰竭风险癌症患者的研究。

AMIA ... Annual Symposium proceedings. AMIA Symposium Pub Date : 2025-05-22 eCollection Date: 2024-01-01

Ziyi Chen, Mengyuan Zhang, Mustafa Mohammed Ahmed, Yi Guo, Thomas J George, Jiang Bian, Yonghui Wu

{"title":"Narrative Feature or Structured Feature? A Study of Large Language Models to Identify Cancer Patients at Risk of Heart Failure.","authors":"Ziyi Chen, Mengyuan Zhang, Mustafa Mohammed Ahmed, Yi Guo, Thomas J George, Jiang Bian, Yonghui Wu","doi":"","DOIUrl":"","url":null,"abstract":"Cancer treatments are known to introduce cardiotoxicity, negatively impacting outcomes and survivorship. Identifying cancer patients at risk of heart failure (HF) is critical to improving cancer treatment outcomes and safety. This study examined machine learning (ML) models to identify cancer patients at risk of HF using electronic health records (EHRs), including traditional ML, Time-Aware long short-term memory (T-LSTM), and large language models (LLMs) using novel narrative features derived from the structured medical codes. We identified a cancer cohort of 12,806 patients from the University of Florida Health, diagnosed with lung, breast, and colorectal cancers, among which 1,602 individuals developed HF after cancer. The LLM, GatorTron-3.9B, achieved the best F1 scores, outperforming the traditional support vector machines by 39%, the T-LSTM deep learning model by 7%, and a widely used transformer model, BERT, by 5.6%. The analysis shows that the proposed narrative features remarkably increased feature density and improved performance.","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"242-251"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12099403/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144144634","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

RealMedQA: A pilot biomedical question answering dataset containing realistic clinical questions. RealMedQA：一个试验性的生物医学问题回答数据集，包含现实的临床问题。

AMIA ... Annual Symposium proceedings. AMIA Symposium Pub Date : 2025-05-22 eCollection Date: 2024-01-01

Gregory Kell, Angus Roberts, Serge Umansky, Yuti Khare, Najma Ahmed, Nikhil Patel, Chloe Simela, Jack Coumbe, Julian Rozario, Ryan-Rhys Griffiths, Iain J Marshall

{"title":"RealMedQA: A pilot biomedical question answering dataset containing realistic clinical questions.","authors":"Gregory Kell, Angus Roberts, Serge Umansky, Yuti Khare, Najma Ahmed, Nikhil Patel, Chloe Simela, Jack Coumbe, Julian Rozario, Ryan-Rhys Griffiths, Iain J Marshall","doi":"","DOIUrl":"","url":null,"abstract":"Clinical question answering systems have the potential to provide clinicians with relevant and timely answers to their questions. Nonetheless, despite the advances that have been made, adoption of these systems in clinical settings has been slow. One issue is a lack of question-answering datasets which reflect the real-world needs of health professionals. In this work, we present RealMedQA, a dataset of realistic clinical questions generated by humans and an LLM. We describe the process for generating and verifying the QA pairs and assess several QA models on BioASQ and RealMedQA to assess the relative difficulty of matching answers to questions. We show that the LLM is more cost-efficient for generating \"ideal\" QA pairs. Additionally, we achieve a lower lexical similarity between questions and answers than BioASQ which provides an additional challenge to the top two QA models, as per the results. We release our code and our dataset publicly to encourage further research.","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"590-599"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12099375/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144144715","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Structured Knowledge Base Enhances Effective Use of Large Language Models for Metadata Curation. 结构化知识库增强了元数据管理中大型语言模型的有效使用。

AMIA ... Annual Symposium proceedings. AMIA Symposium Pub Date : 2025-05-22 eCollection Date: 2024-01-01

Sowmya S Sundaram, Benjamin Solomon, Avani Khatri, Anisha Laumas, Purvesh Khatri, Mark A Musen

{"title":"Structured Knowledge Base Enhances Effective Use of Large Language Models for Metadata Curation.","authors":"Sowmya S Sundaram, Benjamin Solomon, Avani Khatri, Anisha Laumas, Purvesh Khatri, Mark A Musen","doi":"","DOIUrl":"","url":null,"abstract":"Metadata play a crucial role in ensuring the findability, accessibility, interoperability, and reusability of datasets. This paper investigates the potential of large language models (LLMs), specifically GPT-4, to improve adherence to metadata standards in existing datasets. We conducted experiments on 200 random data records describing human samples relating to lung cancer from the NCBI BioSample repository, evaluating GPT-4's ability to suggest edits for adherence to metadata standards. We computed the adherence accuracy of field name-field value pairs through a peer review process, and we observed a marginal average improvement in adherence to the standard data dictionary from 79% to 80% when using GPT-4. We then prompted GPT-4 with domain information in the form of the textual descriptions of CEDAR metadata templates and recorded a statistically significant improvement to 97% from 79% (p<0.01). These results indicate that LLMs show promise for use in automated metadata curation when integrated with a structured knowledge base, though they may struggle when unaided.","PeriodicalId":72180,"journal":{"name":"AMIA ... Annual Symposium proceedings. AMIA Symposium","volume":"2024 ","pages":"1050-1058"},"PeriodicalIF":0.0,"publicationDate":"2025-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12099408/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144144804","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0