Accuracy of an artificial intelligence-enabled diagnostic assistance device in recognizing normal chest radiographs: a service evaluation.

BJR open Pub Date : 2024-09-14 eCollection Date: 2024-01-01 DOI:10.1093/bjro/tzae029
Amrita Kumar, Puja Patel, Dennis Robert, Shamie Kumar, Aneesh Khetani, Bhargava Reddy, Anumeha Srivastava
{"title":"Accuracy of an artificial intelligence-enabled diagnostic assistance device in recognizing normal chest radiographs: a service evaluation.","authors":"Amrita Kumar, Puja Patel, Dennis Robert, Shamie Kumar, Aneesh Khetani, Bhargava Reddy, Anumeha Srivastava","doi":"10.1093/bjro/tzae029","DOIUrl":null,"url":null,"abstract":"<p><strong>Objectives: </strong>Artificial intelligence (AI) enabled devices may be able to optimize radiologists' productivity by identifying normal and abnormal chest X-rays (CXRs) for triaging. In this service evaluation, we investigated the accuracy of one such AI device (qXR).</p><p><strong>Methods: </strong>A randomly sampled subset of general practice and outpatient-referred frontal CXRs from a National Health Service Trust was collected retrospectively from examinations conducted during November 2022 to January 2023. Ground truth was established by consensus between 2 radiologists. The main objective was to estimate negative predictive value (NPV) of AI.</p><p><strong>Results: </strong>A total of 522 CXRs (458 [87.74%] normal CXRs) from 522 patients (median age, 64 years [IQR, 49-77]; 305 [58.43%] female) were analysed. AI predicted 348 CXRs as normal, of which 346 were truly normal (NPV: 99.43% [95% CI, 97.94-99.93]). The sensitivity, specificity, positive predictive value, and area under the ROC curve of AI were found to be 96.88% (95% CI, 89.16-99.62), 75.55% (95% CI, 71.34-79.42), 35.63% (95% CI, 28.53-43.23), and 91.92% (95% CI, 89.38-94.45), respectively. A sensitivity analysis was conducted to estimate NPV by varying assumptions of the prevalence of normal CXRs. The NPV ranged from 88.96% to 99.54% as prevalence increased.</p><p><strong>Conclusions: </strong>The AI device recognized normal CXRs with high NPV and has the potential to increase radiologists' productivity.</p><p><strong>Advances in knowledge: </strong>There is a need for more evidence on the utility of AI-enabled devices in identifying normal CXRs. This work adds to such limited evidence and enables researchers to plan studies to further evaluate the impact of such devices.</p>","PeriodicalId":72419,"journal":{"name":"BJR open","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-09-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11441651/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"BJR open","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1093/bjro/tzae029","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/1/1 0:00:00","PubModel":"eCollection","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

Objectives: Artificial intelligence (AI) enabled devices may be able to optimize radiologists' productivity by identifying normal and abnormal chest X-rays (CXRs) for triaging. In this service evaluation, we investigated the accuracy of one such AI device (qXR).

Methods: A randomly sampled subset of general practice and outpatient-referred frontal CXRs from a National Health Service Trust was collected retrospectively from examinations conducted during November 2022 to January 2023. Ground truth was established by consensus between 2 radiologists. The main objective was to estimate negative predictive value (NPV) of AI.

Results: A total of 522 CXRs (458 [87.74%] normal CXRs) from 522 patients (median age, 64 years [IQR, 49-77]; 305 [58.43%] female) were analysed. AI predicted 348 CXRs as normal, of which 346 were truly normal (NPV: 99.43% [95% CI, 97.94-99.93]). The sensitivity, specificity, positive predictive value, and area under the ROC curve of AI were found to be 96.88% (95% CI, 89.16-99.62), 75.55% (95% CI, 71.34-79.42), 35.63% (95% CI, 28.53-43.23), and 91.92% (95% CI, 89.38-94.45), respectively. A sensitivity analysis was conducted to estimate NPV by varying assumptions of the prevalence of normal CXRs. The NPV ranged from 88.96% to 99.54% as prevalence increased.

Conclusions: The AI device recognized normal CXRs with high NPV and has the potential to increase radiologists' productivity.

Advances in knowledge: There is a need for more evidence on the utility of AI-enabled devices in identifying normal CXRs. This work adds to such limited evidence and enables researchers to plan studies to further evaluate the impact of such devices.

人工智能辅助诊断设备识别正常胸片的准确性:服务评估。
目的:人工智能(AI)设备可以通过识别正常和异常胸部 X 光片(CXR)进行分流,从而优化放射科医生的工作效率。在这项服务评估中,我们调查了一款此类人工智能设备(qXR)的准确性:方法:我们从 2022 年 11 月至 2023 年 1 月期间进行的检查中回顾性地收集了一个国民健康服务信托基金随机抽样的全科和门诊病人转诊的正面 CXR 子集。由两名放射科医生达成共识,确定基本事实。主要目的是估算 AI 的阴性预测值 (NPV):共分析了 522 名患者(中位年龄 64 岁 [IQR,49-77];女性 305 人 [58.43%])的 522 张 CXR(正常 CXR 458 张 [87.74%])。AI 预测 348 例 CXR 为正常,其中 346 例为真正正常(NPV:99.43% [95% CI,97.94-99.93])。AI 的灵敏度、特异性、阳性预测值和 ROC 曲线下面积分别为 96.88%(95% CI,89.16-99.62)、75.55%(95% CI,71.34-79.42)、35.63%(95% CI,28.53-43.23)和 91.92%(95% CI,89.38-94.45)。我们进行了一项敏感性分析,通过不同的 CXR 正常率假设来估算 NPV。随着患病率的增加,NPV 从 88.96% 到 99.54% 不等:结论:人工智能设备识别正常 CXR 的 NPV 很高,具有提高放射医师工作效率的潜力:需要更多证据来证明人工智能设备在识别正常 CXR 方面的效用。这项工作补充了这些有限的证据,使研究人员能够规划研究,进一步评估此类设备的影响。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
审稿时长
18 weeks
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信