Effectiveness of various general large language models in clinical consensus and case analysis in dental implantology: a comparative study.

IF 3.3 3区医学 Q2 MEDICAL INFORMATICS

BMC Medical Informatics and Decision Making Pub Date : 2025-03-26 DOI:10.1186/s12911-025-02972-2

Yuepeng Wu, Yukang Zhang, Mei Xu, Chen Jinzhi, Yican Xue, Yuchen Zheng

{"title":"Effectiveness of various general large language models in clinical consensus and case analysis in dental implantology: a comparative study.","authors":"Yuepeng Wu, Yukang Zhang, Mei Xu, Chen Jinzhi, Yican Xue, Yuchen Zheng","doi":"10.1186/s12911-025-02972-2","DOIUrl":null,"url":null,"abstract":"Background: This study evaluates and compares ChatGPT-4.0, Gemini Pro 1.5(0801), Claude 3 Opus, and Qwen 2.0 72B in answering dental implant questions. The aim is to help doctors in underserved areas choose the best LLMs(Large Language Model) for their procedures, improving dental care accessibility and clinical decision-making.Methods: Two dental implant specialists with over twenty years of clinical experience evaluated the models. Questions were categorized into simple true/false, complex short-answer, and real-life case analyses. Performance was measured using precision, recall, and Bayesian inference-based evaluation metrics.Results: ChatGPT-4 exhibited the most stable and consistent performance on both simple and complex questions. Gemini Pro 1.5(0801)performed well on simple questions but was less stable on complex tasks. Qwen 2.0 72B provided high-quality answers for specific cases but showed variability. Claude 3 opus had the lowest performance across various metrics. Statistical analysis indicated significant differences between models in diagnostic performance but not in treatment planning.Conclusions: ChatGPT-4 is the most reliable model for handling medical questions, followed by Gemini Pro 1.5(0801). Qwen 2.0 72B shows potential but lacks consistency, and Claude 3 Opus performs poorly overall. Combining multiple models is recommended for comprehensive medical decision-making.","PeriodicalId":9340,"journal":{"name":"BMC Medical Informatics and Decision Making","volume":"25 1","pages":"147"},"PeriodicalIF":3.3000,"publicationDate":"2025-03-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11938642/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"BMC Medical Informatics and Decision Making","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1186/s12911-025-02972-2","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"MEDICAL INFORMATICS","Score":null,"Total":0}

引用次数: 0

Abstract

Background: This study evaluates and compares ChatGPT-4.0, Gemini Pro 1.5(0801), Claude 3 Opus, and Qwen 2.0 72B in answering dental implant questions. The aim is to help doctors in underserved areas choose the best LLMs(Large Language Model) for their procedures, improving dental care accessibility and clinical decision-making.

Methods: Two dental implant specialists with over twenty years of clinical experience evaluated the models. Questions were categorized into simple true/false, complex short-answer, and real-life case analyses. Performance was measured using precision, recall, and Bayesian inference-based evaluation metrics.

Results: ChatGPT-4 exhibited the most stable and consistent performance on both simple and complex questions. Gemini Pro 1.5(0801)performed well on simple questions but was less stable on complex tasks. Qwen 2.0 72B provided high-quality answers for specific cases but showed variability. Claude 3 opus had the lowest performance across various metrics. Statistical analysis indicated significant differences between models in diagnostic performance but not in treatment planning.

Conclusions: ChatGPT-4 is the most reliable model for handling medical questions, followed by Gemini Pro 1.5(0801). Qwen 2.0 72B shows potential but lacks consistency, and Claude 3 Opus performs poorly overall. Combining multiple models is recommended for comprehensive medical decision-making.

查看原文本刊更多论文

各种通用大语言模型在种植牙临床共识和病例分析中的有效性比较研究。

背景：本研究对ChatGPT-4.0、Gemini Pro 1.5（0801）、Claude 3 Opus和Qwen 2.0 72B在回答种植牙问题方面进行了评价和比较。目的是帮助服务不足地区的医生为他们的手术选择最好的llm（大型语言模型），提高牙科护理的可及性和临床决策。方法：由两位具有二十多年临床经验的种植专家对模型进行评估。问题分为简单的对/错、复杂的简答和现实案例分析。使用精确度、召回率和基于贝叶斯推理的评估指标来衡量性能。结果：ChatGPT-4在简单和复杂问题上表现出最稳定和一致的表现。Gemini Pro 1.5（0801）在简单问题上表现良好，但在复杂任务上表现不太稳定。Qwen 2.0 72B为特定案例提供了高质量的答案，但存在可变性。Claude 3 opus在各种指标中表现最差。统计分析表明，不同模型在诊断性能上存在显著差异，但在治疗计划上无显著差异。结论：ChatGPT-4是处理医疗问题最可靠的模型，其次是Gemini Pro 1.5（0801）。Qwen 2.0 72B显示出潜力，但缺乏稳定性，Claude 3 Opus整体表现不佳。建议结合多种模型进行综合医疗决策。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

BMC Medical Informatics and Decision Making 医学-医学：信息

CiteScore

7.20

自引率

5.70%

发文量

297

审稿时长

1 months

期刊介绍： BMC Medical Informatics and Decision Making is an open access journal publishing original peer-reviewed research articles in relation to the design, development, implementation, use, and evaluation of health information technologies and decision-making for human health.