Decoding the NCCN Guidelines With AI: A Comparative Evaluation of ChatGPT-4.0 and Llama 2 in the Management of Thyroid Carcinoma.

IF 1 4区医学 Q3 SURGERY

American Surgeon Pub Date : 2025-01-01 Epub Date: 2024-08-13 DOI:10.1177/00031348241269430

Shivam Pandya, Tamir E Bresler, Tyler Wilson, Zin Htway, Manabu Fujita

{"title":"Decoding the NCCN Guidelines With AI: A Comparative Evaluation of ChatGPT-4.0 and Llama 2 in the Management of Thyroid Carcinoma.","authors":"Shivam Pandya, Tamir E Bresler, Tyler Wilson, Zin Htway, Manabu Fujita","doi":"10.1177/00031348241269430","DOIUrl":null,"url":null,"abstract":"Introduction: Artificial Intelligence (AI) has emerged as a promising tool in the delivery of health care. ChatGPT-4.0 (OpenAI, San Francisco, California) and Llama 2 (Meta, Menlo Park, CA) have each gained attention for their use in various medical applications.Objective: This study aims to evaluate and compare the effectiveness of ChatGPT-4.0 and Llama 2 in assisting with complex clinical decision making in the diagnosis and treatment of thyroid carcinoma.Participants: We reviewed the National Comprehensive Cancer Network® (NCCN) Clinical Practice Guidelines for the management of thyroid carcinoma and formulated up to 3 complex clinical questions for each decision-making page. ChatGPT-4.0 and Llama 2 were queried in a reproducible manner. The answers were scored on a Likert scale: 5) Correct; 4) correct, with missing information requiring clarification; 3) correct, but unable to complete answer; 2) partially incorrect; 1) absolutely incorrect. Score frequencies were compared, and subgroup analysis was conducted on Correctness (defined as scores 1-2 vs 3-5) and Accuracy (scores 1-3 vs 4-5).Results: In total, 58 pages of the NCCN Guidelines® were analyzed, generating 167 unique questions. There was no statistically significant difference between ChatGPT-4.0 and Llama 2 in terms of overall score (Mann-Whitney U-test; Mean Rank = 160.53 vs 174.47, P = 0.123), Correctness (P = 0.177), or Accuracy (P = 0.891).[Formula: see text].Conclusion: ChatGPT-4.0 and Llama 2 demonstrate a limited but substantial capacity to assist with complex clinical decision making relating to the management of thyroid carcinoma, with no significant difference in their effectiveness.","PeriodicalId":7782,"journal":{"name":"American Surgeon","volume":" ","pages":"94-98"},"PeriodicalIF":1.0000,"publicationDate":"2025-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"American Surgeon","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1177/00031348241269430","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/8/13 0:00:00","PubModel":"Epub","JCR":"Q3","JCRName":"SURGERY","Score":null,"Total":0}

引用次数: 0

Abstract

Introduction: Artificial Intelligence (AI) has emerged as a promising tool in the delivery of health care. ChatGPT-4.0 (OpenAI, San Francisco, California) and Llama 2 (Meta, Menlo Park, CA) have each gained attention for their use in various medical applications.

Objective: This study aims to evaluate and compare the effectiveness of ChatGPT-4.0 and Llama 2 in assisting with complex clinical decision making in the diagnosis and treatment of thyroid carcinoma.

Participants: We reviewed the National Comprehensive Cancer Network® (NCCN) Clinical Practice Guidelines for the management of thyroid carcinoma and formulated up to 3 complex clinical questions for each decision-making page. ChatGPT-4.0 and Llama 2 were queried in a reproducible manner. The answers were scored on a Likert scale: 5) Correct; 4) correct, with missing information requiring clarification; 3) correct, but unable to complete answer; 2) partially incorrect; 1) absolutely incorrect. Score frequencies were compared, and subgroup analysis was conducted on Correctness (defined as scores 1-2 vs 3-5) and Accuracy (scores 1-3 vs 4-5).

Results: In total, 58 pages of the NCCN Guidelines® were analyzed, generating 167 unique questions. There was no statistically significant difference between ChatGPT-4.0 and Llama 2 in terms of overall score (Mann-Whitney U-test; Mean Rank = 160.53 vs 174.47, P = 0.123), Correctness (P = 0.177), or Accuracy (P = 0.891).[Formula: see text].

Conclusion: ChatGPT-4.0 and Llama 2 demonstrate a limited but substantial capacity to assist with complex clinical decision making relating to the management of thyroid carcinoma, with no significant difference in their effectiveness.

查看原文本刊更多论文

用人工智能解码 NCCN 指南：ChatGPT-4.0 和 Llama 2 在甲状腺癌治疗中的比较评估。

导言：人工智能（AI）已成为提供医疗保健服务的一种前景广阔的工具。ChatGPT-4.0（OpenAI，加利福尼亚州旧金山）和Llama 2（Meta，加利福尼亚州门洛帕克）分别因其在各种医疗应用中的使用而备受关注：本研究旨在评估和比较 ChatGPT-4.0 和 Llama 2 在甲状腺癌诊断和治疗的复杂临床决策中的辅助效果：我们查阅了美国国立综合癌症网络®（NCCN）甲状腺癌临床实践指南，并为每个决策页面制定了多达 3 个复杂的临床问题。以可重复的方式查询了 ChatGPT-4.0 和 Llama 2。答案采用李克特量表评分：5)正确；4)正确，但缺少需要澄清的信息；3)正确，但无法完整回答；2)部分错误；1)完全错误。比较了得分频率，并对正确率（定义为 1-2 分与 3-5 分）和准确率（1-3 分与 4-5 分）进行了分组分析：共分析了 58 页的 NCCN®指南，产生了 167 个独特的问题。ChatGPT-4.0 和 Llama 2 在总分（Mann-Whitney U 检验；平均分 = 160.53 vs 174.47，P = 0.123）、正确率（P = 0.177）或准确率（P = 0.891）方面没有统计学意义上的显著差异：结论：ChatGPT-4.0 和 Llama 2 在甲状腺癌治疗的复杂临床决策方面具有有限但实质性的辅助能力，两者的有效性没有显著差异。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

American Surgeon 医学-外科

CiteScore

1.40

自引率

0.00%

发文量

623

期刊介绍： The American Surgeon is a monthly peer-reviewed publication published by the Southeastern Surgical Congress. Its area of concentration is clinical general surgery, as defined by the content areas of the American Board of Surgery: alimentary tract (including bariatric surgery), abdomen and its contents, breast, skin and soft tissue, endocrine system, solid organ transplantation, pediatric surgery, surgical critical care, surgical oncology (including head and neck surgery), trauma and emergency surgery, and vascular surgery.