A retrospective audit of an artificial intelligence software for the detection of intracranial haemorrhage used by a teleradiology company in the United Kingdom.

BJR open Pub Date : 2024-10-04 eCollection Date: 2024-01-01 DOI:10.1093/bjro/tzae033
Garry Pettet, Julie West, Dennis Robert, Aneesh Khetani, Shamie Kumar, Satish Golla, Robert Lavis
{"title":"A retrospective audit of an artificial intelligence software for the detection of intracranial haemorrhage used by a teleradiology company in the United Kingdom.","authors":"Garry Pettet, Julie West, Dennis Robert, Aneesh Khetani, Shamie Kumar, Satish Golla, Robert Lavis","doi":"10.1093/bjro/tzae033","DOIUrl":null,"url":null,"abstract":"<p><strong>Objectives: </strong>Artificial intelligence (AI) algorithms have the potential to assist radiologists in the reporting of head computed tomography (CT) scans. We investigated the performance of an AI-based software device used in a large teleradiology practice for intracranial haemorrhage (ICH) detection.</p><p><strong>Methods: </strong>A randomly selected subset of all non-contrast CT head (NCCTH) scans from patients aged ≥18 years referred for urgent teleradiology reporting from 44 different hospitals within the United Kingdom over a 4-month period was considered for this evaluation. Thirty auditing radiologists evaluated the NCCTH scans and the AI output retrospectively. Agreement between AI and auditing radiologists is reported along with failure analysis.</p><p><strong>Results: </strong>A total of 1315 NCCTH scans from as many distinct patients (median age, 73 years [IQR 53-84]; 696 [52.9%] females) were evaluated. One hundred twelve (8.5%) scans had ICH. Overall agreement, positive percent agreement, negative percent agreement, and Gwet's AC1 of AI with radiologists were found to be 93.5% (95% CI, 92.1-94.8), 85.7% (77.8-91.6), 94.3% (92.8-95.5) and 0.92 (0.90-0.94), respectively, in detecting ICH. 9 out of 16 false negative outcomes were due to missed subarachnoid haemorrhages and these were predominantly subtle haemorrhages. The most common reason for false positive results was due to motion artefacts.</p><p><strong>Conclusions: </strong>AI demonstrated very good agreement with the radiologists in the detection of ICH.</p><p><strong>Advances in knowledge: </strong>Real-world evaluation of an AI-based CT head interpretation device is reported. Knowledge of scenarios where false negative and false positive results are possible will help reporting radiologists.</p>","PeriodicalId":72419,"journal":{"name":"BJR open","volume":"6 1","pages":"tzae033"},"PeriodicalIF":0.0000,"publicationDate":"2024-10-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11522876/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"BJR open","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1093/bjro/tzae033","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/1/1 0:00:00","PubModel":"eCollection","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

Objectives: Artificial intelligence (AI) algorithms have the potential to assist radiologists in the reporting of head computed tomography (CT) scans. We investigated the performance of an AI-based software device used in a large teleradiology practice for intracranial haemorrhage (ICH) detection.

Methods: A randomly selected subset of all non-contrast CT head (NCCTH) scans from patients aged ≥18 years referred for urgent teleradiology reporting from 44 different hospitals within the United Kingdom over a 4-month period was considered for this evaluation. Thirty auditing radiologists evaluated the NCCTH scans and the AI output retrospectively. Agreement between AI and auditing radiologists is reported along with failure analysis.

Results: A total of 1315 NCCTH scans from as many distinct patients (median age, 73 years [IQR 53-84]; 696 [52.9%] females) were evaluated. One hundred twelve (8.5%) scans had ICH. Overall agreement, positive percent agreement, negative percent agreement, and Gwet's AC1 of AI with radiologists were found to be 93.5% (95% CI, 92.1-94.8), 85.7% (77.8-91.6), 94.3% (92.8-95.5) and 0.92 (0.90-0.94), respectively, in detecting ICH. 9 out of 16 false negative outcomes were due to missed subarachnoid haemorrhages and these were predominantly subtle haemorrhages. The most common reason for false positive results was due to motion artefacts.

Conclusions: AI demonstrated very good agreement with the radiologists in the detection of ICH.

Advances in knowledge: Real-world evaluation of an AI-based CT head interpretation device is reported. Knowledge of scenarios where false negative and false positive results are possible will help reporting radiologists.

对英国一家远程放射学公司使用的颅内出血人工智能检测软件的回顾性审计。
目的:人工智能(AI)算法有可能帮助放射科医生报告头部计算机断层扫描(CT)扫描结果。我们对大型远程放射学实践中用于颅内出血(ICH)检测的基于人工智能的软件设备的性能进行了调查:本次评估随机抽取了英国 44 家不同医院在 4 个月内转诊的年龄≥18 岁的紧急远程放射学报告患者的所有头部非对比 CT(NCCTH)扫描结果。30 位放射审核专家对 NCCTH 扫描和 AI 输出进行了回顾性评估。报告了人工智能与审核放射医师之间的一致性以及失败分析:共评估了来自不同患者(中位年龄 73 岁 [IQR 53-84];女性 696 [52.9%])的 1315 次 NCCTH 扫描。其中 112 例(8.5%)扫描结果为 ICH。发现在检测 ICH 方面,AI 与放射科医生的总体一致率、阳性一致率、阴性一致率和 Gwet's AC1 分别为 93.5% (95% CI, 92.1-94.8)、85.7% (77.8-91.6)、94.3% (92.8-95.5) 和 0.92 (0.90-0.94)。16 个假阴性结果中有 9 个是由于漏诊了蛛网膜下腔出血,这些出血主要是细微出血。造成假阳性结果的最常见原因是运动伪影:结论:在检测 ICH 方面,人工智能与放射科医生表现出很好的一致性:报告对基于人工智能的头部 CT 解释设备进行了真实世界评估。对可能出现假阴性和假阳性结果的情况的了解将有助于报告放射医师。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
审稿时长
18 weeks
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信