{"title":"Accuracy of an artificial intelligence-enabled diagnostic assistance device in recognizing normal chest radiographs: a service evaluation.","authors":"Amrita Kumar, Puja Patel, Dennis Robert, Shamie Kumar, Aneesh Khetani, Bhargava Reddy, Anumeha Srivastava","doi":"10.1093/bjro/tzae029","DOIUrl":null,"url":null,"abstract":"<p><strong>Objectives: </strong>Artificial intelligence (AI) enabled devices may be able to optimize radiologists' productivity by identifying normal and abnormal chest X-rays (CXRs) for triaging. In this service evaluation, we investigated the accuracy of one such AI device (qXR).</p><p><strong>Methods: </strong>A randomly sampled subset of general practice and outpatient-referred frontal CXRs from a National Health Service Trust was collected retrospectively from examinations conducted during November 2022 to January 2023. Ground truth was established by consensus between 2 radiologists. The main objective was to estimate negative predictive value (NPV) of AI.</p><p><strong>Results: </strong>A total of 522 CXRs (458 [87.74%] normal CXRs) from 522 patients (median age, 64 years [IQR, 49-77]; 305 [58.43%] female) were analysed. AI predicted 348 CXRs as normal, of which 346 were truly normal (NPV: 99.43% [95% CI, 97.94-99.93]). The sensitivity, specificity, positive predictive value, and area under the ROC curve of AI were found to be 96.88% (95% CI, 89.16-99.62), 75.55% (95% CI, 71.34-79.42), 35.63% (95% CI, 28.53-43.23), and 91.92% (95% CI, 89.38-94.45), respectively. A sensitivity analysis was conducted to estimate NPV by varying assumptions of the prevalence of normal CXRs. The NPV ranged from 88.96% to 99.54% as prevalence increased.</p><p><strong>Conclusions: </strong>The AI device recognized normal CXRs with high NPV and has the potential to increase radiologists' productivity.</p><p><strong>Advances in knowledge: </strong>There is a need for more evidence on the utility of AI-enabled devices in identifying normal CXRs. This work adds to such limited evidence and enables researchers to plan studies to further evaluate the impact of such devices.</p>","PeriodicalId":72419,"journal":{"name":"BJR open","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-09-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11441651/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"BJR open","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1093/bjro/tzae029","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/1/1 0:00:00","PubModel":"eCollection","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
Objectives: Artificial intelligence (AI) enabled devices may be able to optimize radiologists' productivity by identifying normal and abnormal chest X-rays (CXRs) for triaging. In this service evaluation, we investigated the accuracy of one such AI device (qXR).
Methods: A randomly sampled subset of general practice and outpatient-referred frontal CXRs from a National Health Service Trust was collected retrospectively from examinations conducted during November 2022 to January 2023. Ground truth was established by consensus between 2 radiologists. The main objective was to estimate negative predictive value (NPV) of AI.
Results: A total of 522 CXRs (458 [87.74%] normal CXRs) from 522 patients (median age, 64 years [IQR, 49-77]; 305 [58.43%] female) were analysed. AI predicted 348 CXRs as normal, of which 346 were truly normal (NPV: 99.43% [95% CI, 97.94-99.93]). The sensitivity, specificity, positive predictive value, and area under the ROC curve of AI were found to be 96.88% (95% CI, 89.16-99.62), 75.55% (95% CI, 71.34-79.42), 35.63% (95% CI, 28.53-43.23), and 91.92% (95% CI, 89.38-94.45), respectively. A sensitivity analysis was conducted to estimate NPV by varying assumptions of the prevalence of normal CXRs. The NPV ranged from 88.96% to 99.54% as prevalence increased.
Conclusions: The AI device recognized normal CXRs with high NPV and has the potential to increase radiologists' productivity.
Advances in knowledge: There is a need for more evidence on the utility of AI-enabled devices in identifying normal CXRs. This work adds to such limited evidence and enables researchers to plan studies to further evaluate the impact of such devices.