A systematic review and meta-analysis of diagnostic performance comparison between generative AI and physicians

IF 12.4 1区医学 Q1 HEALTH CARE SCIENCES & SERVICES

NPJ Digital Medicine Pub Date : 2025-03-22 DOI:10.1038/s41746-025-01543-z

Hirotaka Takita, Daijiro Kabata, Shannon L. Walston, Hiroyuki Tatekawa, Kenichi Saito, Yasushi Tsujimoto, Yukio Miki, Daiju Ueda

{"title":"A systematic review and meta-analysis of diagnostic performance comparison between generative AI and physicians","authors":"Hirotaka Takita, Daijiro Kabata, Shannon L. Walston, Hiroyuki Tatekawa, Kenichi Saito, Yasushi Tsujimoto, Yukio Miki, Daiju Ueda","doi":"10.1038/s41746-025-01543-z","DOIUrl":null,"url":null,"abstract":"While generative artificial intelligence (AI) has shown potential in medical diagnostics, comprehensive evaluation of its diagnostic performance and comparison with physicians has not been extensively explored. We conducted a systematic review and meta-analysis of studies validating generative AI models for diagnostic tasks published between June 2018 and June 2024. Analysis of 83 studies revealed an overall diagnostic accuracy of 52.1%. No significant performance difference was found between AI models and physicians overall (p = 0.10) or non-expert physicians (p = 0.93). However, AI models performed significantly worse than expert physicians (p = 0.007). Several models demonstrated slightly higher performance compared to non-experts, although the differences were not significant. Generative AI demonstrates promising diagnostic capabilities with accuracy varying by model. Although it has not yet achieved expert-level reliability, these findings suggest potential for enhancing healthcare delivery and medical education when implemented with appropriate understanding of its limitations.","PeriodicalId":19349,"journal":{"name":"NPJ Digital Medicine","volume":"86 1","pages":""},"PeriodicalIF":12.4000,"publicationDate":"2025-03-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"NPJ Digital Medicine","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1038/s41746-025-01543-z","RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"HEALTH CARE SCIENCES & SERVICES","Score":null,"Total":0}

引用次数: 0

Abstract

While generative artificial intelligence (AI) has shown potential in medical diagnostics, comprehensive evaluation of its diagnostic performance and comparison with physicians has not been extensively explored. We conducted a systematic review and meta-analysis of studies validating generative AI models for diagnostic tasks published between June 2018 and June 2024. Analysis of 83 studies revealed an overall diagnostic accuracy of 52.1%. No significant performance difference was found between AI models and physicians overall (p = 0.10) or non-expert physicians (p = 0.93). However, AI models performed significantly worse than expert physicians (p = 0.007). Several models demonstrated slightly higher performance compared to non-experts, although the differences were not significant. Generative AI demonstrates promising diagnostic capabilities with accuracy varying by model. Although it has not yet achieved expert-level reliability, these findings suggest potential for enhancing healthcare delivery and medical education when implemented with appropriate understanding of its limitations.

Abstract Image

查看原文本刊更多论文

生成式人工智能与医生诊断性能比较的系统回顾和荟萃分析

虽然生成式人工智能（AI）在医疗诊断方面已显示出潜力，但对其诊断性能的全面评估以及与医生的比较尚未得到广泛探讨。我们对2018年6月至2024年6月期间发表的验证生成式人工智能模型诊断任务的研究进行了系统回顾和荟萃分析。对 83 项研究的分析表明，总体诊断准确率为 52.1%。人工智能模型与医生总体（p = 0.10）或非专家医生（p = 0.93）之间没有发现明显的性能差异。不过，人工智能模型的表现明显不如专家医生（p = 0.007）。与非专家相比，有几个模型的性能略高，但差异并不显著。生成式人工智能显示出良好的诊断能力，准确率因模型而异。虽然它尚未达到专家级的可靠性，但这些研究结果表明，在适当了解其局限性的情况下，它有可能改善医疗服务和医学教育。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

NPJ Digital Medicine Multiple-

CiteScore

25.10

自引率

3.30%

发文量

170

审稿时长

15 weeks

期刊介绍： npj Digital Medicine is an online open-access journal that focuses on publishing peer-reviewed research in the field of digital medicine. The journal covers various aspects of digital medicine, including the application and implementation of digital and mobile technologies in clinical settings, virtual healthcare, and the use of artificial intelligence and informatics. The primary goal of the journal is to support innovation and the advancement of healthcare through the integration of new digital and mobile technologies. When determining if a manuscript is suitable for publication, the journal considers four important criteria: novelty, clinical relevance, scientific rigor, and digital innovation.