2025 Expert consensus on retrospective evaluation of large language model applications in clinical scenarios

Abstract

Large language models (LLMs), trained on vast amounts of textual data, have demonstrated strong capabilities in natural language understanding and generation. In the medical field, LLMs are increasingly applied across various domains such as disease screening, diagnostic assistance, and health management, playing a key role in advancing intelligent healthcare. In recent years, China has actively promoted the integration of artificial intelligence (AI) with healthcare through a series of policies that support enterprises in making breakthroughs in key technologies such as medical LLMs and multimodal data integration. Concurrently, efforts have accelerated the deployment of AI in applications such as health management and precision medicine to gradually establish a full-cycle intelligent healthcare system encompassing prevention, diagnosis, treatment, and rehabilitation. However, the rapid deployment of LLMs in healthcare has highlighted the lack of standardized evaluation criteria and consistent methodologies. To address this, this expert consensus focuses on establishing a retrospective evaluation framework tailored to medical applications. By integrating scientific evaluation metrics, standards, and procedures, the framework provides clear and actionable guidance for model evaluators, developers, and end users. It aims to unify assessment practices, enhance the scientific rigor and comparability of evaluations, and ensure the safe and effective use of LLMs in healthcare, ultimately supporting the high-quality development of AI-powered medical services.