AI automated radiographic scoring in rheumatoid arthritis: Shedding light on barriers to implementation through comprehensive evaluation

IF 4.6 2区医学 Q1 RHEUMATOLOGY

Seminars in arthritis and rheumatism Pub Date : 2025-06-05 DOI:10.1016/j.semarthrit.2025.152761

Alix Bird , Lauren Oakden-Rayner , Katrina Chakradeo , Ranjeny Thomas , Drishti Gupta , Suyash Jain , Rohan Jacob , Shonket Ray , Mihir D Wechalekar , Susanna Proudman , Lyle J. Palmer

{"title":"AI automated radiographic scoring in rheumatoid arthritis: Shedding light on barriers to implementation through comprehensive evaluation","authors":"Alix Bird , Lauren Oakden-Rayner , Katrina Chakradeo , Ranjeny Thomas , Drishti Gupta , Suyash Jain , Rohan Jacob , Shonket Ray , Mihir D Wechalekar , Susanna Proudman , Lyle J. Palmer","doi":"10.1016/j.semarthrit.2025.152761","DOIUrl":null,"url":null,"abstract":"<div><h3>Objectives</h3><div>Artificial intelligence (AI) has demonstrated the potential to improve efficiency and reliability of radiographic scoring in rheumatoid arthritis but lacks sufficient evidence to justify clinical use. We developed and rigorously validated a deep learning model to automate radiographic scoring against two external test sets, drawing upon state of the art reporting guidelines to clarify present barriers to implementation.</div></div><div><h3>Methods</h3><div>AI algorithms were trained to predict the Sharp van der Heijde score in hands and feet using a cohort of 157 patients and 1470 radiographs. External replication was undertaken in test datasets from two hospitals (n=253 patients, 589 radiographs). Alongside standard performance metrics to measure error and agreement, we reported subgroup performance, conducted an exploratory analysis of error, and demonstrated relationships with functional outcomes.</div></div><div><h3>Results</h3><div>Our AI system underperformed compared to manual scoring, with lower agreement between the AI and consensus score than between the two manual scorers. The AI system was better at ranking scores than achieving absolute agreement, with intraclass correlation coefficients ranging from 0.03 to 0.27 while Spearman’s correlation coefficients were consistently higher, ranging from 0.16 to 0.55.</div></div><div><h3>Conclusions</h3><div>The performance of the AI systems developed for automating radiographic scoring in RA is insufficient to justify use in research or clinical practice. Large, diverse, and thoroughly described longitudinal datasets will be indispensable in the development and rigorous evaluation of algorithms. Achieving this is key to the ongoing precise evaluation of clinical outcomes in rheumatoid arthritis to enable further improvements to patient care.</div></div>","PeriodicalId":21715,"journal":{"name":"Seminars in arthritis and rheumatism","volume":"74 ","pages":"Article 152761"},"PeriodicalIF":4.6000,"publicationDate":"2025-06-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Seminars in arthritis and rheumatism","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0049017225001325","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"RHEUMATOLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Objectives

Artificial intelligence (AI) has demonstrated the potential to improve efficiency and reliability of radiographic scoring in rheumatoid arthritis but lacks sufficient evidence to justify clinical use. We developed and rigorously validated a deep learning model to automate radiographic scoring against two external test sets, drawing upon state of the art reporting guidelines to clarify present barriers to implementation.

Methods

AI algorithms were trained to predict the Sharp van der Heijde score in hands and feet using a cohort of 157 patients and 1470 radiographs. External replication was undertaken in test datasets from two hospitals (n=253 patients, 589 radiographs). Alongside standard performance metrics to measure error and agreement, we reported subgroup performance, conducted an exploratory analysis of error, and demonstrated relationships with functional outcomes.

Results

Our AI system underperformed compared to manual scoring, with lower agreement between the AI and consensus score than between the two manual scorers. The AI system was better at ranking scores than achieving absolute agreement, with intraclass correlation coefficients ranging from 0.03 to 0.27 while Spearman’s correlation coefficients were consistently higher, ranging from 0.16 to 0.55.

Conclusions

The performance of the AI systems developed for automating radiographic scoring in RA is insufficient to justify use in research or clinical practice. Large, diverse, and thoroughly described longitudinal datasets will be indispensable in the development and rigorous evaluation of algorithms. Achieving this is key to the ongoing precise evaluation of clinical outcomes in rheumatoid arthritis to enable further improvements to patient care.

查看原文本刊更多论文

类风湿关节炎的人工智能自动x线评分：通过综合评估揭示实施障碍

人工智能（AI）已经证明有潜力提高类风湿关节炎放射评分的效率和可靠性，但缺乏足够的证据来证明其临床应用的合理性。我们开发并严格验证了一个深度学习模型，该模型可以针对两个外部测试集自动进行放射成像评分，并利用最先进的报告指南来澄清目前实施的障碍。方法使用157例患者和1470张x线片，训练ai算法预测手足夏普范德海氏评分。对来自两家医院的测试数据集（n=253名患者，589张x线片）进行了外部复制。除了测量误差和一致性的标准性能指标外，我们还报告了子组的性能，进行了误差的探索性分析，并证明了与功能结果的关系。结果与人工评分相比，我们的人工智能系统表现不佳，人工智能和共识评分之间的一致性低于两个人工评分者之间的一致性。人工智能系统更擅长对分数进行排序，而不是实现绝对一致，类内相关系数在0.03到0.27之间，而斯皮尔曼相关系数一直更高，在0.16到0.55之间。结论：用于RA自动放射评分的AI系统的性能不足以证明其在研究或临床实践中的应用。大型、多样、详尽描述的纵向数据集在算法的开发和严格评估中将是不可或缺的。实现这一点是对类风湿关节炎临床结果进行持续精确评估以进一步改善患者护理的关键。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Seminars in arthritis and rheumatism 医学-风湿病学

CiteScore

9.20

自引率

4.00%

发文量

176

审稿时长

46 days

期刊介绍： Seminars in Arthritis and Rheumatism provides access to the highest-quality clinical, therapeutic and translational research about arthritis, rheumatology and musculoskeletal disorders that affect the joints and connective tissue. Each bimonthly issue includes articles giving you the latest diagnostic criteria, consensus statements, systematic reviews and meta-analyses as well as clinical and translational research studies. Read this journal for the latest groundbreaking research and to gain insights from scientists and clinicians on the management and treatment of musculoskeletal and autoimmune rheumatologic diseases. The journal is of interest to rheumatologists, orthopedic surgeons, internal medicine physicians, immunologists and specialists in bone and mineral metabolism.