ChatGPT and Gemini Are Not Consistently Concordant with 2020 AAOS Clinical Practice Guidelines when Evaluating Rotator Cuff Injury.

IF 4.4 1区医学 Q1 ORTHOPEDICS

Arthroscopy-The Journal of Arthroscopic and Related Surgery Pub Date : 2025-02-04 DOI:10.1016/j.arthro.2025.01.039

Michael Megafu, Omar Guerrero, Avanish Yendluri, Bradford O Parsons, Leesa M Galatz, Xinning Li, John D Kelly, Robert L Parisien

{"title":"ChatGPT and Gemini Are Not Consistently Concordant with 2020 AAOS Clinical Practice Guidelines when Evaluating Rotator Cuff Injury.","authors":"Michael Megafu, Omar Guerrero, Avanish Yendluri, Bradford O Parsons, Leesa M Galatz, Xinning Li, John D Kelly, Robert L Parisien","doi":"10.1016/j.arthro.2025.01.039","DOIUrl":null,"url":null,"abstract":"Purpose: The objective of this study was to evaluate the accuracy of suggestions given by ChatGPT, developed by Open AI and Gemini (previously known as Bard), the two widely used publicly available language learning models (LLMs), to evaluate the management of rotator cuff injuries.Methods: The 2020 AAOS Clinical Practice Guidelines (CPG) were the basis for determining recommended and non-recommended treatments in this study. ChatGPT and Gemini were queried on 16 treatments based on these guidelines examining rotator cuff interventions. These responses were categorized as \"Concordant\" or \"Discordant\" with the AAOS CPG. A Cohen's Kappa coefficient was calculated to assess the interrater reliability.Results: Of the 16 treatments queried, ChatGPT and Gemini demonstrated concordance with the AAOS CPG for 13 (81%) and 12 (75%) treatments, respectively. ChatGPT provided discordant responses with the AAOS CPG for three treatments (19%), while Gemini provided discordant responses for four treatments (25%). Using Cohen's Kappa coefficient, the interrater reliability was 0.98 between the raters, signifying agreement in classifying the responses of ChatGPT and Gemini to the AAOS CPG as being \"concordant\" or \"discordant.\"Conclusion: ChatGPT and Gemini do not consistently provide responses that align with the AAOS CPG.","PeriodicalId":55459,"journal":{"name":"Arthroscopy-The Journal of Arthroscopic and Related Surgery","volume":" ","pages":""},"PeriodicalIF":4.4000,"publicationDate":"2025-02-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Arthroscopy-The Journal of Arthroscopic and Related Surgery","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1016/j.arthro.2025.01.039","RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ORTHOPEDICS","Score":null,"Total":0}

引用次数: 0

Abstract

Purpose: The objective of this study was to evaluate the accuracy of suggestions given by ChatGPT, developed by Open AI and Gemini (previously known as Bard), the two widely used publicly available language learning models (LLMs), to evaluate the management of rotator cuff injuries.

Methods: The 2020 AAOS Clinical Practice Guidelines (CPG) were the basis for determining recommended and non-recommended treatments in this study. ChatGPT and Gemini were queried on 16 treatments based on these guidelines examining rotator cuff interventions. These responses were categorized as "Concordant" or "Discordant" with the AAOS CPG. A Cohen's Kappa coefficient was calculated to assess the interrater reliability.

Results: Of the 16 treatments queried, ChatGPT and Gemini demonstrated concordance with the AAOS CPG for 13 (81%) and 12 (75%) treatments, respectively. ChatGPT provided discordant responses with the AAOS CPG for three treatments (19%), while Gemini provided discordant responses for four treatments (25%). Using Cohen's Kappa coefficient, the interrater reliability was 0.98 between the raters, signifying agreement in classifying the responses of ChatGPT and Gemini to the AAOS CPG as being "concordant" or "discordant."

Conclusion: ChatGPT and Gemini do not consistently provide responses that align with the AAOS CPG.

查看原文本刊更多论文

求助全文

约1分钟内获得全文求助全文

来源期刊

Arthroscopy-The Journal of Arthroscopic and Related Surgery 医学-外科

CiteScore

9.30

自引率

17.00%

发文量

555

审稿时长

58 days

期刊介绍： Nowhere is minimally invasive surgery explained better than in Arthroscopy, the leading peer-reviewed journal in the field. Every issue enables you to put into perspective the usefulness of the various emerging arthroscopic techniques. The advantages and disadvantages of these methods -- along with their applications in various situations -- are discussed in relation to their efficiency, efficacy and cost benefit. As a special incentive, paid subscribers also receive access to the journal expanded website.