Caution Regarding ChatGPT's Appropriateness and Reliability Regarding Surgery for Wrist Arthritis.

IF 1.8 Q2 ORTHOPEDICS
HAND Pub Date : 2025-09-01 Epub Date: 2024-07-24 DOI:10.1177/15589447241265519
Keegan Hones, Emily Krisanda, Harvey Chim
{"title":"Caution Regarding ChatGPT's Appropriateness and Reliability Regarding Surgery for Wrist Arthritis.","authors":"Keegan Hones, Emily Krisanda, Harvey Chim","doi":"10.1177/15589447241265519","DOIUrl":null,"url":null,"abstract":"<p><strong>Background: </strong>Chat Generative Pre-Trained Transformer (ChatGPT), an artificial intelligence (AI) program, is widely used for information compilation. This study sought to analyze the quality and consistency of the information generated by ChatGPT regarding common procedures for wrist arthritis.</p><p><strong>Methods: </strong>32 standardized questions regarding wrist osteoarthritis and related procedures (4-corner-fusion [4CF], proximal row carpectomy [PRC], resurfacing capitate pyrocarbon implant, wrist denervation, and total wrist arthrodesis and arthroplasty) were presented to the ChatGPT-3.5 interface 3 separate times, without feedback. ChatGPT's answers were evaluated for medical accuracy by 3 reviewers and rated as \"appropriate,\" \"appropriate but incomplete,\" or \"inappropriate.\" Ratings were then converted to numerical values to calculate an intraclass correlation coefficient (ICC). A DISCERN score was used to assess quality, and Flesch-Kincade Grade Level and Flesch Reading Ease Score for readability.</p><p><strong>Results: </strong>75% of the responses were deemed \"appropriate,\" with 23 questions receiving unanimous appropriate ratings across all responses. The ICC was 0.97 (95% CI [0.46, 0.98]), indicating excellent reliability. DISCERN score was 60 (good). The Flesch-Kincaid Grade Level was 14.6 ± 1.9, and the Flesch Reading Ease Score was 25.3 ± 6.7, implying a college reading level. The information that ChatGPT provided for PRC and total wrist arthrodesis and arthroplasty, appeared to be more reliable than for 4CF and denervation.</p><p><strong>Conclusion: </strong>ChatGPT's reliability and accuracy of information varied across procedures, possibly due to unknown and diverse sources. Furthermore, while some answers were factually correct, many provided generic information across differing questions, limiting usefulness. ChatGPT must be used cautiously, and the limitations understood.</p>","PeriodicalId":12902,"journal":{"name":"HAND","volume":" ","pages":"910-916"},"PeriodicalIF":1.8000,"publicationDate":"2025-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11571340/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"HAND","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1177/15589447241265519","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/7/24 0:00:00","PubModel":"Epub","JCR":"Q2","JCRName":"ORTHOPEDICS","Score":null,"Total":0}
引用次数: 0

Abstract

Background: Chat Generative Pre-Trained Transformer (ChatGPT), an artificial intelligence (AI) program, is widely used for information compilation. This study sought to analyze the quality and consistency of the information generated by ChatGPT regarding common procedures for wrist arthritis.

Methods: 32 standardized questions regarding wrist osteoarthritis and related procedures (4-corner-fusion [4CF], proximal row carpectomy [PRC], resurfacing capitate pyrocarbon implant, wrist denervation, and total wrist arthrodesis and arthroplasty) were presented to the ChatGPT-3.5 interface 3 separate times, without feedback. ChatGPT's answers were evaluated for medical accuracy by 3 reviewers and rated as "appropriate," "appropriate but incomplete," or "inappropriate." Ratings were then converted to numerical values to calculate an intraclass correlation coefficient (ICC). A DISCERN score was used to assess quality, and Flesch-Kincade Grade Level and Flesch Reading Ease Score for readability.

Results: 75% of the responses were deemed "appropriate," with 23 questions receiving unanimous appropriate ratings across all responses. The ICC was 0.97 (95% CI [0.46, 0.98]), indicating excellent reliability. DISCERN score was 60 (good). The Flesch-Kincaid Grade Level was 14.6 ± 1.9, and the Flesch Reading Ease Score was 25.3 ± 6.7, implying a college reading level. The information that ChatGPT provided for PRC and total wrist arthrodesis and arthroplasty, appeared to be more reliable than for 4CF and denervation.

Conclusion: ChatGPT's reliability and accuracy of information varied across procedures, possibly due to unknown and diverse sources. Furthermore, while some answers were factually correct, many provided generic information across differing questions, limiting usefulness. ChatGPT must be used cautiously, and the limitations understood.

注意 ChatGPT 对腕关节炎手术的适宜性和可靠性。
背景介绍聊天生成预训练转换器(ChatGPT)是一种人工智能(AI)程序,被广泛用于信息编译。本研究旨在分析 ChatGPT 生成的有关腕关节炎常见手术的信息的质量和一致性。方法:在没有反馈的情况下,向 ChatGPT-3.5 界面提交 32 个有关腕关节骨关节炎及相关手术(4 角融合术 [4CF]、近端行腕骨切除术 [PRC]、重铺头状热碳植入物、腕关节去神经化、全腕关节置换术和关节置换术)的标准化问题。ChatGPT 的答案由 3 位审查员进行医学准确性评估,并评定为 "适当"、"适当但不完整 "或 "不适当"。然后将评分转换成数值,计算类内相关系数 (ICC)。DISCERN 评分用于评估质量,Flesch-Kincade 等级评分和 Flesch 阅读容易程度评分用于评估可读性:75%的回答被认为是 "适当的",其中有 23 个问题在所有回答中获得了一致的 "适当 "评价。ICC 为 0.97(95% CI [0.46,0.98]),显示出极佳的可靠性。DISCERN 得分为 60 分(良好)。弗莱什-金凯德年级水平(Flesch-Kincaid Grade Level)为 14.6 ± 1.9,弗莱什阅读轻松度得分(Flesch Reading Ease Score)为 25.3 ± 6.7,这意味着学生的阅读水平达到了大学水平。与 4CF 和去神经支配相比,ChatGPT 为 PRC 和全腕关节置换术提供的信息似乎更可靠:结论:ChatGPT 信息的可靠性和准确性因手术而异,这可能是由于信息来源不明且多种多样。此外,虽然有些答案与事实相符,但许多答案提供的是不同问题的通用信息,从而限制了其实用性。必须谨慎使用 ChatGPT 并了解其局限性。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
HAND
HAND Medicine-Surgery
CiteScore
3.30
自引率
0.00%
发文量
209
期刊介绍: HAND is the official journal of the American Association for Hand Surgery and is a peer-reviewed journal featuring articles written by clinicians worldwide presenting current research and clinical work in the field of hand surgery. It features articles related to all aspects of hand and upper extremity surgery and the post operative care and rehabilitation of the hand.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信