Rapid review: Growing usage of Multimodal Large Language Models in healthcare

IF 4.5 2区医学 Q2 COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS

Journal of Biomedical Informatics Pub Date : 2025-08-05 DOI:10.1016/j.jbi.2025.104875

Pallavi Gupta , Zhihong Zhang , Meijia Song , Martin Michalowski , Xiao Hu , Gregor Stiglic , Maxim Topaz

{"title":"Rapid review: Growing usage of Multimodal Large Language Models in healthcare","authors":"Pallavi Gupta , Zhihong Zhang , Meijia Song , Martin Michalowski , Xiao Hu , Gregor Stiglic , Maxim Topaz","doi":"10.1016/j.jbi.2025.104875","DOIUrl":null,"url":null,"abstract":"<div><h3>Objective:</h3><div>Recent advancements in large language models (LLMs) have led to multimodal LLMs (MLLMs), which integrate multiple data modalities beyond text. Although MLLMs show promise, there is a gap in the literature that empirically demonstrates their impact in healthcare. This paper summarizes the applications of MLLMs in healthcare, highlighting their potential to transform health practices.</div></div><div><h3>Methods:</h3><div>A rapid literature review was conducted in August 2024 using World Health Organization (WHO) rapid-review methodology and PRISMA standards, with searches across four databases (Scopus, Medline, PubMed and ACM Digital Library) and top-tier conferences—including NeurIPS, ICML, AAAI, MICCAI, CVPR, ACL and EMNLP. Articles on MLLMs healthcare applications were included for analysis based on inclusion and exclusion criteria.</div></div><div><h3>Results:</h3><div>The search yielded 115 articles, 39 included in the final analysis. Of these, 77% appeared online (preprints and published) in 2024, reflecting the emergence of MLLMs. 80% of studies were from Asia and North America (mainly China and US), with Europe lagging. Studies split evenly between pre-built MLLMs evaluations (60% focused on GPT versions) and custom MLLMs/frameworks development with task-specific customizations. About 81% of studies examined MLLMs for diagnosis and reporting in radiology, pathology, and ophthalmology, with additional applications in education, surgery, and mental health. Prompting strategies, used in 80% of studies, improved performance in nearly half. However, evaluation practices were inconsistent with 67% reported accuracy. Error analysis was mostly anecdotal, with only 18% categorized failure types. Only 13% validated explainability through clinician feedback. Clinical deployment was demonstrated in just 3% of studies, and workflow integration, governance, and safety were rarely addressed.</div></div><div><h3>Discussion and Conclusion:</h3><div>MLLMs offer substantial potential for healthcare transformation through multimodal data integration. Yet, methodological inconsistencies, limited validation, and underdeveloped deployment strategies highlight the need for standardized evaluation metrics, structured error analysis, and human-centered design to support safe, scalable, and trustworthy clinical adoption.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"169 ","pages":"Article 104875"},"PeriodicalIF":4.5000,"publicationDate":"2025-08-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Biomedical Informatics","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1532046425001042","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS","Score":null,"Total":0}

引用次数: 0

Abstract

Objective:

Recent advancements in large language models (LLMs) have led to multimodal LLMs (MLLMs), which integrate multiple data modalities beyond text. Although MLLMs show promise, there is a gap in the literature that empirically demonstrates their impact in healthcare. This paper summarizes the applications of MLLMs in healthcare, highlighting their potential to transform health practices.

Methods:

A rapid literature review was conducted in August 2024 using World Health Organization (WHO) rapid-review methodology and PRISMA standards, with searches across four databases (Scopus, Medline, PubMed and ACM Digital Library) and top-tier conferences—including NeurIPS, ICML, AAAI, MICCAI, CVPR, ACL and EMNLP. Articles on MLLMs healthcare applications were included for analysis based on inclusion and exclusion criteria.

Results:

The search yielded 115 articles, 39 included in the final analysis. Of these, 77% appeared online (preprints and published) in 2024, reflecting the emergence of MLLMs. 80% of studies were from Asia and North America (mainly China and US), with Europe lagging. Studies split evenly between pre-built MLLMs evaluations (60% focused on GPT versions) and custom MLLMs/frameworks development with task-specific customizations. About 81% of studies examined MLLMs for diagnosis and reporting in radiology, pathology, and ophthalmology, with additional applications in education, surgery, and mental health. Prompting strategies, used in 80% of studies, improved performance in nearly half. However, evaluation practices were inconsistent with 67% reported accuracy. Error analysis was mostly anecdotal, with only 18% categorized failure types. Only 13% validated explainability through clinician feedback. Clinical deployment was demonstrated in just 3% of studies, and workflow integration, governance, and safety were rarely addressed.

Discussion and Conclusion:

MLLMs offer substantial potential for healthcare transformation through multimodal data integration. Yet, methodological inconsistencies, limited validation, and underdeveloped deployment strategies highlight the need for standardized evaluation metrics, structured error analysis, and human-centered design to support safe, scalable, and trustworthy clinical adoption.

Abstract Image

查看原文本刊更多论文

快速回顾：多模态大语言模型在医疗保健中的日益增长的使用

目的：大型语言模型（llm）的最新进展导致了多模态llm (mllm)，它集成了文本以外的多种数据模式。虽然传销显示出希望，有一个差距，在文献经验证明其在医疗保健的影响。本文总结了mllm在医疗保健中的应用，强调了它们改变医疗实践的潜力。方法：于2024年8月采用世界卫生组织（WHO）快速综述方法和PRISMA标准，检索4个数据库（Scopus、Medline、PubMed和ACM数字图书馆）和顶级会议（包括NeurIPS、ICML、AAAI、MICCAI、CVPR、ACL和EMNLP）进行快速文献综述。根据纳入和排除标准，纳入了关于mllm医疗保健应用的文章进行分析。结果：检索到115篇文章，其中39篇纳入最终分析。其中，77%在2024年出现在网上（预印本和出版），反映了传销的出现。80%的研究来自亚洲和北美（主要是中国和美国），欧洲落后。研究平均分为预构建的mllm评估（60%集中在GPT版本）和带有特定任务定制的定制mllm /框架开发。大约81%的研究检查了mllm在放射学、病理学和眼科学中的诊断和报告，并在教育、外科和心理健康方面有额外的应用。在80%的研究中使用的提示策略提高了近一半的表现。然而，评估实践与67%报告的准确性不一致。错误分析主要是轶事，只有18%的故障类型被分类。只有13%通过临床医生反馈验证了可解释性。临床部署仅在3%的研究中得到证实，工作流程集成、治理和安全性很少得到解决。讨论和结论：mllm通过多模式数据集成为医疗保健转型提供了巨大的潜力。然而，方法上的不一致、有限的验证和不发达的部署策略突出了对标准化评估指标、结构化错误分析和以人为本的设计的需求，以支持安全、可扩展和可信赖的临床采用。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Biomedical Informatics 医学-计算机：跨学科应用

CiteScore

8.90

自引率

6.70%

发文量

243

审稿时长

32 days

期刊介绍： The Journal of Biomedical Informatics reflects a commitment to high-quality original research papers, reviews, and commentaries in the area of biomedical informatics methodology. Although we publish articles motivated by applications in the biomedical sciences (for example, clinical medicine, health care, population health, and translational bioinformatics), the journal emphasizes reports of new methodologies and techniques that have general applicability and that form the basis for the evolving science of biomedical informatics. Articles on medical devices; evaluations of implemented systems (including clinical trials of information technologies); or papers that provide insight into a biological process, a specific disease, or treatment options would generally be more suitable for publication in other venues. Papers on applications of signal processing and image analysis are often more suitable for biomedical engineering journals or other informatics journals, although we do publish papers that emphasize the information management and knowledge representation/modeling issues that arise in the storage and use of biological signals and images. System descriptions are welcome if they illustrate and substantiate the underlying methodology that is the principal focus of the report and an effort is made to address the generalizability and/or range of application of that methodology. Note also that, given the international nature of JBI, papers that deal with specific languages other than English, or with country-specific health systems or approaches, are acceptable for JBI only if they offer generalizable lessons that are relevant to the broad JBI readership, regardless of their country, language, culture, or health system.