Suhasini Gupta, Brett D Haislup, Anisha Tyagi, Suleiman Y Sudah, Ryan A Hoffman, Anand M Murthi
{"title":"Assessment and comparison of artificial intelligence-generated information regarding shoulder arthroplasty from multiple interfaces.","authors":"Suhasini Gupta, Brett D Haislup, Anisha Tyagi, Suleiman Y Sudah, Ryan A Hoffman, Anand M Murthi","doi":"10.1016/j.jse.2024.12.048","DOIUrl":null,"url":null,"abstract":"<p><strong>Background: </strong>This study aims to analyze and compare the quality, accuracy, and readability of information regarding anatomic total shoulder arthroplasty (aTSA) and reverse total shoulder arthroplasty (rTSA) provided by various AI interfaces (Open AI's ChatGPT and Microsoft's CoPilot).</p><p><strong>Methods: </strong>Thirty commonly asked questions (categorized by Rothwell criteria into Fact, Policy, and Value) by patients were inputted into ChatGPT 3.5 and CoPilot. Responses were assessed with the DISCERN scale, Journal of the American Medical Association (JAMA) benchmark criteria, and Flesch-Kincaid Reading Ease Score (FRES) and Flesch-Kincaid Grade Level (FKGL). The sources of citations provided by CoPilot were further analyzed.</p><p><strong>Results: </strong>Both AI interfaces generated DISCERN scores >50 (aTSA and rTSA ChatGPT: 57 [Fact], 61 [Policy], 58 [Value]; aTSA and rTSA CoPilot: 68 [Fact], 72 [Policy], 70 [Value]), demonstrating \"good\" quality of information provided, except for the Policy questions by CoPilot, which were scored as \"excellent\" (>70). CoPilot's higher JAMA score (3 vs. 0) and FRES scores >30 indicated more reliable, accessible responses, which required a minimum of 12th-grade education to read the same. In comparison, the ChatGPT generated more complex texts, with the majority of the FRES scores <20, and FKGL score signifying complexity of academic level text. Finally, CoPilot provided citations and demonstrated the highest percentage of academic sources (31.1% for rTSA and 26.7% for aTSA), suggesting reliable sources of information.</p><p><strong>Conclusion: </strong>Overall, the information provided by both AI interfaces ChatGPT and CoPilot was scored as a \"good\" source of information for commonly asked patient questions regarding shoulder arthroplasty. But the answers to questions pertaining to shoulder arthroplasty provided by CoPilot proved to be more reliable (P = .0061), less complex, easier to read (P = .0031), and referenced information from reliable resources including academic sources, journal articles, and medical sites. Although answers provided by CoPilot were \"easier\" to read, they still required a 12th-grade education, which may be too complex for most patients, posing a challenge for patient comprehension. There were a substantial amount of nonmedical media sites, and commercial sources that were cited for both aTSA and rTSA questions by CoPilot. Critically, answers from both AI interfaces should serve as supplementary resources rather than primary sources on perioperative conditions pertaining to shoulder arthroplasty.</p>","PeriodicalId":50051,"journal":{"name":"Journal of Shoulder and Elbow Surgery","volume":" ","pages":""},"PeriodicalIF":2.9000,"publicationDate":"2025-02-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Shoulder and Elbow Surgery","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1016/j.jse.2024.12.048","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ORTHOPEDICS","Score":null,"Total":0}
引用次数: 0
Abstract
Background: This study aims to analyze and compare the quality, accuracy, and readability of information regarding anatomic total shoulder arthroplasty (aTSA) and reverse total shoulder arthroplasty (rTSA) provided by various AI interfaces (Open AI's ChatGPT and Microsoft's CoPilot).
Methods: Thirty commonly asked questions (categorized by Rothwell criteria into Fact, Policy, and Value) by patients were inputted into ChatGPT 3.5 and CoPilot. Responses were assessed with the DISCERN scale, Journal of the American Medical Association (JAMA) benchmark criteria, and Flesch-Kincaid Reading Ease Score (FRES) and Flesch-Kincaid Grade Level (FKGL). The sources of citations provided by CoPilot were further analyzed.
Results: Both AI interfaces generated DISCERN scores >50 (aTSA and rTSA ChatGPT: 57 [Fact], 61 [Policy], 58 [Value]; aTSA and rTSA CoPilot: 68 [Fact], 72 [Policy], 70 [Value]), demonstrating "good" quality of information provided, except for the Policy questions by CoPilot, which were scored as "excellent" (>70). CoPilot's higher JAMA score (3 vs. 0) and FRES scores >30 indicated more reliable, accessible responses, which required a minimum of 12th-grade education to read the same. In comparison, the ChatGPT generated more complex texts, with the majority of the FRES scores <20, and FKGL score signifying complexity of academic level text. Finally, CoPilot provided citations and demonstrated the highest percentage of academic sources (31.1% for rTSA and 26.7% for aTSA), suggesting reliable sources of information.
Conclusion: Overall, the information provided by both AI interfaces ChatGPT and CoPilot was scored as a "good" source of information for commonly asked patient questions regarding shoulder arthroplasty. But the answers to questions pertaining to shoulder arthroplasty provided by CoPilot proved to be more reliable (P = .0061), less complex, easier to read (P = .0031), and referenced information from reliable resources including academic sources, journal articles, and medical sites. Although answers provided by CoPilot were "easier" to read, they still required a 12th-grade education, which may be too complex for most patients, posing a challenge for patient comprehension. There were a substantial amount of nonmedical media sites, and commercial sources that were cited for both aTSA and rTSA questions by CoPilot. Critically, answers from both AI interfaces should serve as supplementary resources rather than primary sources on perioperative conditions pertaining to shoulder arthroplasty.
期刊介绍：
The official publication for eight leading specialty organizations, this authoritative journal is the only publication to focus exclusively on medical, surgical, and physical techniques for treating injury/disease of the upper extremity, including the shoulder girdle, arm, and elbow. Clinically oriented and peer-reviewed, the Journal provides an international forum for the exchange of information on new techniques, instruments, and materials. Journal of Shoulder and Elbow Surgery features vivid photos, professional illustrations, and explicit diagrams that demonstrate surgical approaches and depict implant devices. Topics covered include fractures, dislocations, diseases and injuries of the rotator cuff, imaging techniques, arthritis, arthroscopy, arthroplasty, and rehabilitation.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
