Consensus on the Potential of Large Language Models in Healthcare: Insights from a Delphi Survey in Korea.

Abstract

Objectives: Given the rapidly growing expectations for large language models (LLMs) in healthcare, this study systematically collected perspectives from Korean experts on the potential benefits and risks of LLMs, aiming to promote their safe and effective utilization.

Methods: A web-based mini-Delphi survey was conducted from August 27 to October 14, 2024, with 20 selected panelists. The expert questionnaire comprised 84 judgment items across five domains: potential applications, benefits, risks, reliability requirements, and safe usage. These items were developed through a literature review and expert consultation. Participants rated their agreement or perceived importance on a 5-point scale. Items meeting predefined thresholds (content validity ratio ≥0.49, degree of convergence ≤0.50, and degree of consensus ≥0.75) were prioritized.

Results: Seventeen participants (85%) responded to the first round, and 16 participants (80%) completed the second round. Consensus was achieved on several potential applications, benefits, and reliability requirements for the use of LLMs in healthcare. However, significant heterogeneity was found regarding perceptions of associated risks and criteria for safe usage of LLMs. Of the 84 total items, 52 met the criteria for statistical validity, confirming the diversity of expert opinions.

Conclusions: Experts reached a consensus on certain aspects of LLM utilization in healthcare. Nonetheless, notable differences remained concerning risks and requirements for safe implementation, highlighting the need for further investigation. This study provides foundational insights to guide future research and inform policy development for the responsible introduction of LLMs into the healthcare field.