Evaluating artificial intelligence performance in medical image analysis: Sensitivity, specificity, accuracy, and precision of ChatGPT-4o on Kellgren-Lawrence grading of knee X-ray radiographs

IF 1.6 4区医学 Q3 ORTHOPEDICS

Knee Pub Date : 2025-04-23 DOI:10.1016/j.knee.2025.04.008

Mustafa Hüseyin Temel , Yakup Erden , Fatih Bağcıer

{"title":"Evaluating artificial intelligence performance in medical image analysis: Sensitivity, specificity, accuracy, and precision of ChatGPT-4o on Kellgren-Lawrence grading of knee X-ray radiographs","authors":"Mustafa Hüseyin Temel , Yakup Erden , Fatih Bağcıer","doi":"10.1016/j.knee.2025.04.008","DOIUrl":null,"url":null,"abstract":"<div><h3>Background</h3><div>Recent advancements in artificial intelligence, including ChatGPT, have enabled its application in medical image analysis.This study aimed to evaluate the sensitivity and specificity of ChatGPT in assessing knee osteoarthritis (KOA) radiographs using the Kellgren-Lawrence (KL) grading system.</div></div><div><h3>Methods</h3><div>A retrospective study was conducted at Izzet Baysal Physical Therapy and Rehabilitation Training and Research Hospital. Anteroposterior weight-bearing knee X-rays from 226 patients (excluding 26 due to prostheses or foreign bodies) were evaluated. Two specialists assessed the radiographs using the KL grading system, with a third specialist resolving discrepancies. ChatGPT-4o evaluated the images using the prompt, “Please evaluate this knee anteroposterior radiographic image according to the Kellgren-Lawrence grading system.” Diagnostic accuracy metrics, receiver operating characteristic (ROC) curves, and area under the curve (AUC) values were calculated.</div></div><div><h3>Results</h3><div>ChatGPT showed low sensitivity across all grades. The accuracy of the model was calculated to be 0.230. ROC AUC values were low for all grades, for KL grade 0 at 0.53, KL grade 1 at 0.56, KL grade 2 at 0.43, KL grade 3 at 0.54, KL grade 4 at 0.49, micro-average at 0.52, macro-average at 0.51, and weighted average at 0.52.</div></div><div><h3>Conclusions</h3><div>The findings of this study highlight the model’s inability to reliably distinguish between KL grades, suggesting that its utility in this specific classification task is limited and requires further optimization to improve its predictive accuracy and reliability. The model’s current limitations preclude its use as a reliable diagnostic tool. Further refinement is necessary to improve its clinical applicability.</div></div>","PeriodicalId":56110,"journal":{"name":"Knee","volume":"55 ","pages":"Pages 79-84"},"PeriodicalIF":1.6000,"publicationDate":"2025-04-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Knee","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0968016025000791","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"ORTHOPEDICS","Score":null,"Total":0}

引用次数: 0

Abstract

Background

Recent advancements in artificial intelligence, including ChatGPT, have enabled its application in medical image analysis.This study aimed to evaluate the sensitivity and specificity of ChatGPT in assessing knee osteoarthritis (KOA) radiographs using the Kellgren-Lawrence (KL) grading system.

Methods

A retrospective study was conducted at Izzet Baysal Physical Therapy and Rehabilitation Training and Research Hospital. Anteroposterior weight-bearing knee X-rays from 226 patients (excluding 26 due to prostheses or foreign bodies) were evaluated. Two specialists assessed the radiographs using the KL grading system, with a third specialist resolving discrepancies. ChatGPT-4o evaluated the images using the prompt, “Please evaluate this knee anteroposterior radiographic image according to the Kellgren-Lawrence grading system.” Diagnostic accuracy metrics, receiver operating characteristic (ROC) curves, and area under the curve (AUC) values were calculated.

Results

ChatGPT showed low sensitivity across all grades. The accuracy of the model was calculated to be 0.230. ROC AUC values were low for all grades, for KL grade 0 at 0.53, KL grade 1 at 0.56, KL grade 2 at 0.43, KL grade 3 at 0.54, KL grade 4 at 0.49, micro-average at 0.52, macro-average at 0.51, and weighted average at 0.52.

Conclusions

The findings of this study highlight the model’s inability to reliably distinguish between KL grades, suggesting that its utility in this specific classification task is limited and requires further optimization to improve its predictive accuracy and reliability. The model’s current limitations preclude its use as a reliable diagnostic tool. Further refinement is necessary to improve its clinical applicability.

查看原文本刊更多论文

评估人工智能在医学图像分析中的表现：chatgpt - 40在膝关节x线片kelgren - lawrence分级中的敏感性、特异性、准确性和精密度

人工智能的最新进展，包括ChatGPT，使其在医学图像分析中的应用成为可能。本研究旨在利用Kellgren-Lawrence （KL）分级系统评估ChatGPT在评估膝关节骨关节炎（KOA） x线片中的敏感性和特异性。方法在伊泽特贝萨尔物理治疗与康复训练研究医院进行回顾性研究。对226例患者（不包括26例假体或异物）的负重膝关节前后位x光片进行评估。两位专家使用KL分级系统评估x线片，第三位专家解决差异。chatgpt - 40使用提示来评估图像，“请根据Kellgren-Lawrence分级系统评估此膝关节正位x线图像。”计算诊断准确度指标、受试者工作特征（ROC）曲线和曲线下面积（AUC）值。结果schatgpt在各年级的敏感性均较低。模型的计算精度为0.230。所有等级的ROC AUC值都很低，KL 0级为0.53，KL 1级为0.56，KL 2级为0.43，KL 3级为0.54，KL 4级为0.49，微观平均为0.52，宏观平均为0.51，加权平均为0.52。本研究的发现突出了该模型无法可靠地区分KL等级，表明其在此特定分类任务中的效用有限，需要进一步优化以提高其预测准确性和可靠性。该模型目前的局限性使其无法作为可靠的诊断工具使用。为提高其临床适用性，还需进一步细化。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Knee 医学-外科

CiteScore

3.80

自引率

5.30%

发文量

171

审稿时长

6 months

期刊介绍： The Knee is an international journal publishing studies on the clinical treatment and fundamental biomechanical characteristics of this joint. The aim of the journal is to provide a vehicle relevant to surgeons, biomedical engineers, imaging specialists, materials scientists, rehabilitation personnel and all those with an interest in the knee. The topics covered include, but are not limited to: • Anatomy, physiology, morphology and biochemistry; • Biomechanical studies; • Advances in the development of prosthetic, orthotic and augmentation devices; • Imaging and diagnostic techniques; • Pathology; • Trauma; • Surgery; • Rehabilitation.