B-119 Comparative performance of GPT-4 and CNV-ETLAI in extracting copy number variations from medical journals: Bridging the gap between large language models and specialized NLP tools in genomic data interpretation

IF 7.1 2区医学 Q1 MEDICAL LABORATORY TECHNOLOGY

Clinical chemistry Pub Date : 2024-10-02 DOI:10.1093/clinchem/hvae106.480

J Choi

{"title":"B-119 Comparative performance of GPT-4 and CNV-ETLAI in extracting copy number variations from medical journals: Bridging the gap between large language models and specialized NLP tools in genomic data interpretation","authors":"J Choi","doi":"10.1093/clinchem/hvae106.480","DOIUrl":null,"url":null,"abstract":"Background Copy Number Variations (CNVs) are critical genetic markers in diversity and disease, yet their accurate extraction from medical literature remains challenging due to the complexity of genetic data. While specialized NLP models like CNV-ETLAI have been developed for this task, the advent of Large Language Models (LLMs) such as GPT-4 presents a potential alternative with broader applicability. This study evaluates the efficacy of GPT-4 against CNV-ETLAI in extracting CNVs from medical journal articles, aiming to enhance genetic research and clinical decision-making. Methods We configured GPT-4 to process and interpret medical journal PDFs, designing custom prompts for CNV information extraction. The performance of GPT-4 was benchmarked against CNV-ETLAI using a dataset of 146 true positive CNVs extracted from 23 journal articles. Performance metrics focused on accuracy in extracting CNVs from both text and tables, recognizing the importance of structured data interpretation in genomic analysis. Results CNV-ETLAI demonstrated superior accuracy, achieving a 98% success rate in CNV extraction, compared to GPT-4’s 49%. Specifically, CNV-ETLAI outperformed GPT-4 in table extraction accuracy (99% vs. 41.2%) and context extraction accuracy (96% vs. 63.2%). Despite GPT-4's lower performance, its capacity for improvement and adaptability was noted, indicating potential future applicability in medical data extraction. Conclusions The study highlights CNV-ETLAI's current superiority in extracting CNVs from medical texts, particularly in interpreting structured data like tables. However, the adaptability and potential for growth in LLMs like GPT-4 suggest they could soon become valuable tools for medical data extraction, offering a more versatile and powerful solution across a broader range of applications. The promise of LLMs, despite their current limitations, underscores the need for continued research and development in AI technologies for genomic data interpretation.","PeriodicalId":10690,"journal":{"name":"Clinical chemistry","volume":"191 1","pages":""},"PeriodicalIF":7.1000,"publicationDate":"2024-10-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Clinical chemistry","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1093/clinchem/hvae106.480","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"MEDICAL LABORATORY TECHNOLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Background Copy Number Variations (CNVs) are critical genetic markers in diversity and disease, yet their accurate extraction from medical literature remains challenging due to the complexity of genetic data. While specialized NLP models like CNV-ETLAI have been developed for this task, the advent of Large Language Models (LLMs) such as GPT-4 presents a potential alternative with broader applicability. This study evaluates the efficacy of GPT-4 against CNV-ETLAI in extracting CNVs from medical journal articles, aiming to enhance genetic research and clinical decision-making. Methods We configured GPT-4 to process and interpret medical journal PDFs, designing custom prompts for CNV information extraction. The performance of GPT-4 was benchmarked against CNV-ETLAI using a dataset of 146 true positive CNVs extracted from 23 journal articles. Performance metrics focused on accuracy in extracting CNVs from both text and tables, recognizing the importance of structured data interpretation in genomic analysis. Results CNV-ETLAI demonstrated superior accuracy, achieving a 98% success rate in CNV extraction, compared to GPT-4’s 49%. Specifically, CNV-ETLAI outperformed GPT-4 in table extraction accuracy (99% vs. 41.2%) and context extraction accuracy (96% vs. 63.2%). Despite GPT-4's lower performance, its capacity for improvement and adaptability was noted, indicating potential future applicability in medical data extraction. Conclusions The study highlights CNV-ETLAI's current superiority in extracting CNVs from medical texts, particularly in interpreting structured data like tables. However, the adaptability and potential for growth in LLMs like GPT-4 suggest they could soon become valuable tools for medical data extraction, offering a more versatile and powerful solution across a broader range of applications. The promise of LLMs, despite their current limitations, underscores the need for continued research and development in AI technologies for genomic data interpretation.

查看原文本刊更多论文

B-119 GPT-4 和 CNV-ETLAI 从医学期刊中提取拷贝数变异的性能比较：缩小基因组数据解读中大型语言模型与专业 NLP 工具之间的差距

背景拷贝数变异（CNV）是多样性和疾病中的重要遗传标记，但由于遗传数据的复杂性，从医学文献中准确提取这些变异仍然具有挑战性。虽然已经开发了专门的 NLP 模型（如 CNV-ETAI）来完成这项任务，但大语言模型（LLM）（如 GPT-4）的出现提供了一种具有更广泛适用性的潜在替代方案。本研究评估了 GPT-4 与 CNV-ETLAI 在从医学期刊文章中提取 CNV 方面的功效，旨在加强遗传学研究和临床决策。方法我们配置了 GPT-4 来处理和解释医学期刊 PDF，并设计了用于 CNV 信息提取的自定义提示。我们使用从 23 篇期刊论文中提取的 146 个真阳性 CNV 数据集，将 GPT-4 的性能与 CNV-ETLAI 进行了比较。性能指标主要集中在从文本和表格中提取 CNV 的准确性上，这充分体现了结构化数据解释在基因组分析中的重要性。结果 CNV-ETLAI 的准确性更胜一筹，CNV 提取成功率高达 98%，而 GPT-4 的成功率仅为 49%。具体来说，CNV-ETAI 在表格提取准确率（99% 对 41.2%）和上下文提取准确率（96% 对 63.2%）方面均优于 GPT-4。尽管 GPT-4 的性能较低，但其改进能力和适应性受到关注，这表明其未来在医疗数据提取方面具有潜在的适用性。结论该研究强调了 CNV-ETLAI 目前在从医学文本中提取 CNV 方面的优势，尤其是在解释表格等结构化数据方面。不过，GPT-4 等 LLM 的适应性和发展潜力表明，它们很快就会成为医学数据提取的重要工具，为更广泛的应用提供更通用、更强大的解决方案。尽管 LLM 目前还存在局限性，但其前景广阔，这凸显了继续研究和开发用于基因组数据解读的人工智能技术的必要性。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Clinical chemistry 医学-医学实验技术

CiteScore

11.30

自引率

4.30%

发文量

212

审稿时长

1.7 months

期刊介绍： Clinical Chemistry is a peer-reviewed scientific journal that is the premier publication for the science and practice of clinical laboratory medicine. It was established in 1955 and is associated with the Association for Diagnostics & Laboratory Medicine (ADLM). The journal focuses on laboratory diagnosis and management of patients, and has expanded to include other clinical laboratory disciplines such as genomics, hematology, microbiology, and toxicology. It also publishes articles relevant to clinical specialties including cardiology, endocrinology, gastroenterology, genetics, immunology, infectious diseases, maternal-fetal medicine, neurology, nutrition, oncology, and pediatrics. In addition to original research, editorials, and reviews, Clinical Chemistry features recurring sections such as clinical case studies, perspectives, podcasts, and Q&A articles. It has the highest impact factor among journals of clinical chemistry, laboratory medicine, pathology, analytical chemistry, transfusion medicine, and clinical microbiology. The journal is indexed in databases such as MEDLINE and Web of Science.