Jiesheng Zhu, Yilun Jiang, Daosen Chen, Yi Lu, Yijiang Huang, Yimu Lin, Pei Fan
{"title":"ChatGPT-4o 对膝关节骨性关节炎 X 光片的识别率和正负鉴别力较高，但详细分级的准确性有限。","authors":"Jiesheng Zhu, Yilun Jiang, Daosen Chen, Yi Lu, Yijiang Huang, Yimu Lin, Pei Fan","doi":"10.1002/ksa.12639","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Purpose</h3>\n \n <p>To explore the potential of ChatGPT-4o in analysing radiographic images of knee osteoarthritis (OA) and to assess its grading accuracy, feature identification and reliability, thereby helping surgeons to improve diagnostic accuracy and efficiency.</p>\n </section>\n \n <section>\n \n <h3> Methods</h3>\n \n <p>A total of 117 anterior‒posterior knee radiographs from patients (23.1% men, 76.9% women, mean age 69.7 ± 7.99 years) were analysed. Two senior orthopaedic surgeons and ChatGPT-4o independently graded images with the Kellgren–Lawrence (K–L), Ahlbäck and International Knee Documentation Committee (IKDC) systems. A consensus reference standard was established by a third radiologist. ChatGPT-4o's performance metrics (accuracy, precision, recall and F1 score) were calculated, and its reliability was assessed via two evaluations separated by a 2-week interval, with intraclass correlation coefficients (ICCs) determined.</p>\n </section>\n \n <section>\n \n <h3> Results</h3>\n \n <p>ChatGPT-4o achieved a 100% identification rate for knee radiographs and demonstrated strong binary classification performance (precision: 0.95, recall: 0.83, F score: 0.88). However, its detailed grading accuracy (35%) was substantially lower than that of surgeons (89.6%). Severe underestimation of OA severity occurred in 49.3% of the cases. Interrater reliability for surgeons was excellent (ICC: 0.78–0.91), whereas ChatGPT-4o showed poor initial consistency (ICC: 0.16–0.28), improving marginally in the second evaluation (ICC: 0.22–0.39).</p>\n </section>\n \n <section>\n \n <h3> Conclusion</h3>\n \n <p>ChatGPT-4o has the potential to rapidly identify and binary classify knee OA on radiographs. However, its detailed grading accuracy remains suboptimal, with a notable tendency to underestimate severe cases. This limits its current clinical utility for precise staging. Future research should focus on optimising its grading performance and improving accuracy to enhance diagnostic reliability.</p>\n </section>\n \n <section>\n \n <h3> level of Evidence</h3>\n \n <p>Level III, retrospective comparative study.</p>\n </section>\n </div>","PeriodicalId":17880,"journal":{"name":"Knee Surgery, Sports Traumatology, Arthroscopy","volume":"33 5","pages":"1911-1919"},"PeriodicalIF":3.3000,"publicationDate":"2025-03-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"High identification and positive-negative discrimination but limited detailed grading accuracy of ChatGPT-4o in knee osteoarthritis radiographs\",\"authors\":\"Jiesheng Zhu, Yilun Jiang, Daosen Chen, Yi Lu, Yijiang Huang, Yimu Lin, Pei Fan\",\"doi\":\"10.1002/ksa.12639\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div>\\n \\n \\n <section>\\n \\n <h3> Purpose</h3>\\n \\n <p>To explore the potential of ChatGPT-4o in analysing radiographic images of knee osteoarthritis (OA) and to assess its grading accuracy, feature identification and reliability, thereby helping surgeons to improve diagnostic accuracy and efficiency.</p>\\n </section>\\n \\n <section>\\n \\n <h3> Methods</h3>\\n \\n <p>A total of 117 anterior‒posterior knee radiographs from patients (23.1% men, 76.9% women, mean age 69.7 ± 7.99 years) were analysed. Two senior orthopaedic surgeons and ChatGPT-4o independently graded images with the Kellgren–Lawrence (K–L), Ahlbäck and International Knee Documentation Committee (IKDC) systems. A consensus reference standard was established by a third radiologist. ChatGPT-4o's performance metrics (accuracy, precision, recall and F1 score) were calculated, and its reliability was assessed via two evaluations separated by a 2-week interval, with intraclass correlation coefficients (ICCs) determined.</p>\\n </section>\\n \\n <section>\\n \\n <h3> Results</h3>\\n \\n <p>ChatGPT-4o achieved a 100% identification rate for knee radiographs and demonstrated strong binary classification performance (precision: 0.95, recall: 0.83, F score: 0.88). However, its detailed grading accuracy (35%) was substantially lower than that of surgeons (89.6%). Severe underestimation of OA severity occurred in 49.3% of the cases. Interrater reliability for surgeons was excellent (ICC: 0.78–0.91), whereas ChatGPT-4o showed poor initial consistency (ICC: 0.16–0.28), improving marginally in the second evaluation (ICC: 0.22–0.39).</p>\\n </section>\\n \\n <section>\\n \\n <h3> Conclusion</h3>\\n \\n <p>ChatGPT-4o has the potential to rapidly identify and binary classify knee OA on radiographs. However, its detailed grading accuracy remains suboptimal, with a notable tendency to underestimate severe cases. This limits its current clinical utility for precise staging. Future research should focus on optimising its grading performance and improving accuracy to enhance diagnostic reliability.</p>\\n </section>\\n \\n <section>\\n \\n <h3> level of Evidence</h3>\\n \\n <p>Level III, retrospective comparative study.</p>\\n </section>\\n </div>\",\"PeriodicalId\":17880,\"journal\":{\"name\":\"Knee Surgery, Sports Traumatology, Arthroscopy\",\"volume\":\"33 5\",\"pages\":\"1911-1919\"},\"PeriodicalIF\":3.3000,\"publicationDate\":\"2025-03-07\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Knee Surgery, Sports Traumatology, Arthroscopy\",\"FirstCategoryId\":\"3\",\"ListUrlMain\":\"https://onlinelibrary.wiley.com/doi/10.1002/ksa.12639\",\"RegionNum\":2,\"RegionCategory\":\"医学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"ORTHOPEDICS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Knee Surgery, Sports Traumatology, Arthroscopy","FirstCategoryId":"3","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1002/ksa.12639","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ORTHOPEDICS","Score":null,"Total":0}
High identification and positive-negative discrimination but limited detailed grading accuracy of ChatGPT-4o in knee osteoarthritis radiographs
Purpose
To explore the potential of ChatGPT-4o in analysing radiographic images of knee osteoarthritis (OA) and to assess its grading accuracy, feature identification and reliability, thereby helping surgeons to improve diagnostic accuracy and efficiency.
Methods
A total of 117 anterior‒posterior knee radiographs from patients (23.1% men, 76.9% women, mean age 69.7 ± 7.99 years) were analysed. Two senior orthopaedic surgeons and ChatGPT-4o independently graded images with the Kellgren–Lawrence (K–L), Ahlbäck and International Knee Documentation Committee (IKDC) systems. A consensus reference standard was established by a third radiologist. ChatGPT-4o's performance metrics (accuracy, precision, recall and F1 score) were calculated, and its reliability was assessed via two evaluations separated by a 2-week interval, with intraclass correlation coefficients (ICCs) determined.
Results
ChatGPT-4o achieved a 100% identification rate for knee radiographs and demonstrated strong binary classification performance (precision: 0.95, recall: 0.83, F score: 0.88). However, its detailed grading accuracy (35%) was substantially lower than that of surgeons (89.6%). Severe underestimation of OA severity occurred in 49.3% of the cases. Interrater reliability for surgeons was excellent (ICC: 0.78–0.91), whereas ChatGPT-4o showed poor initial consistency (ICC: 0.16–0.28), improving marginally in the second evaluation (ICC: 0.22–0.39).
Conclusion
ChatGPT-4o has the potential to rapidly identify and binary classify knee OA on radiographs. However, its detailed grading accuracy remains suboptimal, with a notable tendency to underestimate severe cases. This limits its current clinical utility for precise staging. Future research should focus on optimising its grading performance and improving accuracy to enhance diagnostic reliability.
期刊介绍：
Few other areas of orthopedic surgery and traumatology have undergone such a dramatic evolution in the last 10 years as knee surgery, arthroscopy and sports traumatology. Ranked among the top 33% of journals in both Orthopedics and Sports Sciences, the goal of this European journal is to publish papers about innovative knee surgery, sports trauma surgery and arthroscopy. Each issue features a series of peer-reviewed articles that deal with diagnosis and management and with basic research. Each issue also contains at least one review article about an important clinical problem. Case presentations or short notes about technical innovations are also accepted for publication.
The articles cover all aspects of knee surgery and all types of sports trauma; in addition, epidemiology, diagnosis, treatment and prevention, and all types of arthroscopy (not only the knee but also the shoulder, elbow, wrist, hip, ankle, etc.) are addressed. Articles on new diagnostic techniques such as MRI and ultrasound and high-quality articles about the biomechanics of joints, muscles and tendons are included. Although this is largely a clinical journal, it is also open to basic research with clinical relevance.
Because the journal is supported by a distinguished European Editorial Board, assisted by an international Advisory Board, you can be assured that the journal maintains the highest standards.
Official Clinical Journal of the European Society of Sports Traumatology, Knee Surgery and Arthroscopy (ESSKA).
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
