系统性文献综述和荟萃分析:适用于 ICH 和 LVO 检测的人工智能应用在真实世界和临床验证中的表现

Jason Le , Oisín Butler , Ann-Kathrin Frenz , Ankur Sharma
{"title":"系统性文献综述和荟萃分析:适用于 ICH 和 LVO 检测的人工智能应用在真实世界和临床验证中的表现","authors":"Jason Le ,&nbsp;Oisín Butler ,&nbsp;Ann-Kathrin Frenz ,&nbsp;Ankur Sharma","doi":"10.1016/j.ibmed.2024.100187","DOIUrl":null,"url":null,"abstract":"<div><h3>Purpose</h3><div>We sought to compare the performance of AI applications in real-world studies to validation study data used to gain regulatory approval.</div></div><div><h3>Methods</h3><div>We searched PubMed, EBSCO, and EMBASE for publications from 2018 to 2023. We included articles that evaluated the sensitivity and specificity of ICH and LVO detection applications in real-world populations. We performed a quality and applicability assessment using QUADAS-2. We used a bivariate or two univariate meta-analyses, where appropriate, to calculate summary point estimates for sensitivity and specificity.</div></div><div><h3>Results</h3><div>Eighteen articles met the criteria of the systematic literature review. The included articles evaluated five applications indicated for ICH or LVO triage. Three of the five applications yielded adequate studies to be included in the meta-analysis. For most applications, we did not observe any systematic differences in sensitivity and specificity results between the point estimates from the meta-analysis and the respective 510k studies. For VIZ LVO and RAPID LVO, the 95 % CI for real-world sensitivity sat within the 95 % CI from their respective validation study. For BriefCase ICH, the 95 % CI for real-world sensitivity sat below the 95 % CI of the respective validation study. Additionally, the 95 % CI for real-world specificity for all three of the applications sat within the 95 % CI of their respective validation studies. Data from the individual real-world studies for RAPID ICH and CINA LVO followed a similar trend.</div></div><div><h3>Conclusion</h3><div>The performance of applications in real-world settings was non-inferior to the performance observed in validation studies used to obtain 510k clearance.</div></div>","PeriodicalId":73399,"journal":{"name":"Intelligence-based medicine","volume":"10 ","pages":"Article 100187"},"PeriodicalIF":0.0000,"publicationDate":"2024-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Systematic literature review and meta-analysis for real-world versus clinical validation performance of artificial intelligence applications indicated for ICH and LVO detection\",\"authors\":\"Jason Le ,&nbsp;Oisín Butler ,&nbsp;Ann-Kathrin Frenz ,&nbsp;Ankur Sharma\",\"doi\":\"10.1016/j.ibmed.2024.100187\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><h3>Purpose</h3><div>We sought to compare the performance of AI applications in real-world studies to validation study data used to gain regulatory approval.</div></div><div><h3>Methods</h3><div>We searched PubMed, EBSCO, and EMBASE for publications from 2018 to 2023. We included articles that evaluated the sensitivity and specificity of ICH and LVO detection applications in real-world populations. We performed a quality and applicability assessment using QUADAS-2. We used a bivariate or two univariate meta-analyses, where appropriate, to calculate summary point estimates for sensitivity and specificity.</div></div><div><h3>Results</h3><div>Eighteen articles met the criteria of the systematic literature review. The included articles evaluated five applications indicated for ICH or LVO triage. Three of the five applications yielded adequate studies to be included in the meta-analysis. For most applications, we did not observe any systematic differences in sensitivity and specificity results between the point estimates from the meta-analysis and the respective 510k studies. For VIZ LVO and RAPID LVO, the 95 % CI for real-world sensitivity sat within the 95 % CI from their respective validation study. For BriefCase ICH, the 95 % CI for real-world sensitivity sat below the 95 % CI of the respective validation study. Additionally, the 95 % CI for real-world specificity for all three of the applications sat within the 95 % CI of their respective validation studies. Data from the individual real-world studies for RAPID ICH and CINA LVO followed a similar trend.</div></div><div><h3>Conclusion</h3><div>The performance of applications in real-world settings was non-inferior to the performance observed in validation studies used to obtain 510k clearance.</div></div>\",\"PeriodicalId\":73399,\"journal\":{\"name\":\"Intelligence-based medicine\",\"volume\":\"10 \",\"pages\":\"Article 100187\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-01-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Intelligence-based medicine\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S2666521224000541\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Intelligence-based medicine","FirstCategoryId":"1085","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2666521224000541","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

摘要

目的我们试图比较人工智能应用在真实世界研究中的表现与用于获得监管部门批准的验证研究数据。方法我们检索了PubMed、EBSCO和EMBASE上2018年至2023年的出版物。我们纳入了评估真实世界人群中 ICH 和 LVO 检测应用灵敏度和特异性的文章。我们使用 QUADAS-2 进行了质量和适用性评估。我们酌情使用双变量或两个单变量荟萃分析来计算灵敏度和特异性的汇总点估计值。纳入的文章评估了五种用于 ICH 或 LVO 分流的应用。在这五种应用中,有三种应用的研究结果足以纳入荟萃分析。对于大多数应用,我们没有观察到荟萃分析的点估计值与相应的 510k 研究之间在灵敏度和特异性结果上存在任何系统性差异。对于 VIZ LVO 和 RAPID LVO,真实世界灵敏度的 95 % CI 位于各自验证研究的 95 % CI 范围内。对于 BriefCase ICH,实际灵敏度的 95 % CI 低于各自验证研究的 95 % CI。此外,所有三种应用的实际特异性的 95 % CI 都在各自验证研究的 95 % CI 范围内。RAPID ICH 和 CINA LVO 的单项真实世界研究数据也呈现类似趋势。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
Systematic literature review and meta-analysis for real-world versus clinical validation performance of artificial intelligence applications indicated for ICH and LVO detection

Purpose

We sought to compare the performance of AI applications in real-world studies to validation study data used to gain regulatory approval.

Methods

We searched PubMed, EBSCO, and EMBASE for publications from 2018 to 2023. We included articles that evaluated the sensitivity and specificity of ICH and LVO detection applications in real-world populations. We performed a quality and applicability assessment using QUADAS-2. We used a bivariate or two univariate meta-analyses, where appropriate, to calculate summary point estimates for sensitivity and specificity.

Results

Eighteen articles met the criteria of the systematic literature review. The included articles evaluated five applications indicated for ICH or LVO triage. Three of the five applications yielded adequate studies to be included in the meta-analysis. For most applications, we did not observe any systematic differences in sensitivity and specificity results between the point estimates from the meta-analysis and the respective 510k studies. For VIZ LVO and RAPID LVO, the 95 % CI for real-world sensitivity sat within the 95 % CI from their respective validation study. For BriefCase ICH, the 95 % CI for real-world sensitivity sat below the 95 % CI of the respective validation study. Additionally, the 95 % CI for real-world specificity for all three of the applications sat within the 95 % CI of their respective validation studies. Data from the individual real-world studies for RAPID ICH and CINA LVO followed a similar trend.

Conclusion

The performance of applications in real-world settings was non-inferior to the performance observed in validation studies used to obtain 510k clearance.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
Intelligence-based medicine
Intelligence-based medicine Health Informatics
CiteScore
5.00
自引率
0.00%
发文量
0
审稿时长
187 days
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信