Margot M. Rakers, Marieke M. van Buchem, Sergej Kucenko, Anne de Hond, Ilse Kant, Maarten van Smeden, Karel G. M. Moons, Artuur M. Leeuwenberg, Niels Chavannes, María Villalobos-Quesada, Hendrikus J. A. van Os
{"title":"Availability of Evidence for Predictive Machine Learning Algorithms in Primary Care","authors":"Margot M. Rakers, Marieke M. van Buchem, Sergej Kucenko, Anne de Hond, Ilse Kant, Maarten van Smeden, Karel G. M. Moons, Artuur M. Leeuwenberg, Niels Chavannes, María Villalobos-Quesada, Hendrikus J. A. van Os","doi":"10.1001/jamanetworkopen.2024.32990","DOIUrl":null,"url":null,"abstract":"ImportanceThe aging and multimorbid population and health personnel shortages pose a substantial burden on primary health care. While predictive machine learning (ML) algorithms have the potential to address these challenges, concerns include transparency and insufficient reporting of model validation and effectiveness of the implementation in the clinical workflow.ObjectivesTo systematically identify predictive ML algorithms implemented in primary care from peer-reviewed literature and US Food and Drug Administration (FDA) and Conformité Européene (CE) registration databases and to ascertain the public availability of evidence, including peer-reviewed literature, gray literature, and technical reports across the artificial intelligence (AI) life cycle.Evidence ReviewPubMed, Embase, Web of Science, Cochrane Library, Emcare, Academic Search Premier, IEEE Xplore, ACM Digital Library, MathSciNet, AAAI.org (Association for the Advancement of Artificial Intelligence), arXiv, Epistemonikos, PsycINFO, and Google Scholar were searched for studies published between January 2000 and July 2023, with search terms that were related to AI, primary care, and implementation. The search extended to CE-marked or FDA-approved predictive ML algorithms obtained from relevant registration databases. Three reviewers gathered subsequent evidence involving strategies such as product searches, exploration of references, manufacturer website visits, and direct inquiries to authors and product owners. The extent to which the evidence for each predictive ML algorithm aligned with the Dutch AI predictive algorithm (AIPA) guideline requirements was assessed per AI life cycle phase, producing evidence availability scores.FindingsThe systematic search identified 43 predictive ML algorithms, of which 25 were commercially available and CE-marked or FDA-approved. The predictive ML algorithms spanned multiple clinical domains, but most (27 [63%]) focused on cardiovascular diseases and diabetes. Most (35 [81%]) were published within the past 5 years. The availability of evidence varied across different phases of the predictive ML algorithm life cycle, with evidence being reported the least for phase 1 (preparation) and phase 5 (impact assessment) (19% and 30%, respectively). Twelve (28%) predictive ML algorithms achieved approximately half of their maximum individual evidence availability score. Overall, predictive ML algorithms from peer-reviewed literature showed higher evidence availability compared with those from FDA-approved or CE-marked databases (45% vs 29%).Conclusions and RelevanceThe findings indicate an urgent need to improve the availability of evidence regarding the predictive ML algorithms’ quality criteria. Adopting the Dutch AIPA guideline could facilitate transparent and consistent reporting of the quality criteria that could foster trust among end users and facilitating large-scale implementation.","PeriodicalId":14694,"journal":{"name":"JAMA Network Open","volume":null,"pages":null},"PeriodicalIF":10.5000,"publicationDate":"2024-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"JAMA Network Open","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1001/jamanetworkopen.2024.32990","RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"MEDICINE, GENERAL & INTERNAL","Score":null,"Total":0}
引用次数: 0
Abstract
ImportanceThe aging and multimorbid population and health personnel shortages pose a substantial burden on primary health care. While predictive machine learning (ML) algorithms have the potential to address these challenges, concerns include transparency and insufficient reporting of model validation and effectiveness of the implementation in the clinical workflow.ObjectivesTo systematically identify predictive ML algorithms implemented in primary care from peer-reviewed literature and US Food and Drug Administration (FDA) and Conformité Européene (CE) registration databases and to ascertain the public availability of evidence, including peer-reviewed literature, gray literature, and technical reports across the artificial intelligence (AI) life cycle.Evidence ReviewPubMed, Embase, Web of Science, Cochrane Library, Emcare, Academic Search Premier, IEEE Xplore, ACM Digital Library, MathSciNet, AAAI.org (Association for the Advancement of Artificial Intelligence), arXiv, Epistemonikos, PsycINFO, and Google Scholar were searched for studies published between January 2000 and July 2023, with search terms that were related to AI, primary care, and implementation. The search extended to CE-marked or FDA-approved predictive ML algorithms obtained from relevant registration databases. Three reviewers gathered subsequent evidence involving strategies such as product searches, exploration of references, manufacturer website visits, and direct inquiries to authors and product owners. The extent to which the evidence for each predictive ML algorithm aligned with the Dutch AI predictive algorithm (AIPA) guideline requirements was assessed per AI life cycle phase, producing evidence availability scores.FindingsThe systematic search identified 43 predictive ML algorithms, of which 25 were commercially available and CE-marked or FDA-approved. The predictive ML algorithms spanned multiple clinical domains, but most (27 [63%]) focused on cardiovascular diseases and diabetes. Most (35 [81%]) were published within the past 5 years. The availability of evidence varied across different phases of the predictive ML algorithm life cycle, with evidence being reported the least for phase 1 (preparation) and phase 5 (impact assessment) (19% and 30%, respectively). Twelve (28%) predictive ML algorithms achieved approximately half of their maximum individual evidence availability score. Overall, predictive ML algorithms from peer-reviewed literature showed higher evidence availability compared with those from FDA-approved or CE-marked databases (45% vs 29%).Conclusions and RelevanceThe findings indicate an urgent need to improve the availability of evidence regarding the predictive ML algorithms’ quality criteria. Adopting the Dutch AIPA guideline could facilitate transparent and consistent reporting of the quality criteria that could foster trust among end users and facilitating large-scale implementation.
重要性老龄化和多病人口以及医务人员短缺给初级医疗保健带来了沉重负担。目标从同行评议文献、美国食品药品管理局(FDA)和欧洲合格评定委员会(CE)注册数据库中系统地识别在初级医疗中实施的预测性机器学习(ML)算法,并确定证据的公开可用性,包括同行评议文献、灰色文献和人工智能(AI)生命周期中的技术报告。证据回顾PubMed、Embase、Web of Science、Cochrane Library、Emcare、Academic Search Premier、IEEE Xplore、ACM Digital Library、MathSciNet、AAAI.org (Association for the Advancement of Artificial Intelligence)、arXiv、Epistemonikos、PsycINFO和Google Scholar检索了2000年1月至2023年7月期间发表的研究,检索词与人工智能、初级保健和实施相关。搜索范围还包括从相关注册数据库中获得的 CE 标记或 FDA 批准的预测性 ML 算法。三位审稿人通过产品搜索、查阅参考文献、访问制造商网站以及直接询问作者和产品所有者等策略收集后续证据。每个人工智能生命周期阶段对每种预测性人工智能算法的证据符合荷兰人工智能预测算法(AIPA)指南要求的程度进行评估,得出证据可用性分数。这些预测性 ML 算法涉及多个临床领域,但大多数(27 [63%])侧重于心血管疾病和糖尿病。大多数算法(35 [81%])是在过去 5 年内发表的。在预测性 ML 算法生命周期的不同阶段,证据的可用性各不相同,第 1 阶段(准备)和第 5 阶段(影响评估)报告的证据最少(分别为 19% 和 30%)。有 12 种(28%)预测性 ML 算法的证据可用性达到了其单项最高分的一半左右。总体而言,来自同行评议文献的预测性 ML 算法的证据可用性高于来自 FDA 批准或 CE 标识数据库的算法(45% vs 29%)。采用荷兰 AIPA 准则可促进质量标准的透明化和一致性报告,从而增强最终用户的信任度,促进大规模实施。
期刊介绍:
JAMA Network Open, a member of the esteemed JAMA Network, stands as an international, peer-reviewed, open-access general medical journal.The publication is dedicated to disseminating research across various health disciplines and countries, encompassing clinical care, innovation in health care, health policy, and global health.
JAMA Network Open caters to clinicians, investigators, and policymakers, providing a platform for valuable insights and advancements in the medical field. As part of the JAMA Network, a consortium of peer-reviewed general medical and specialty publications, JAMA Network Open contributes to the collective knowledge and understanding within the medical community.