Beyond multiple-choice questions: Rethinking evaluation frameworks for large language models for clinical medicine

Abstract

Large language models (LLMs) have demonstrated encouraging performance for medical natural language processing (NLP) tasks, approaching human-equivalent performance in some of the standard benchmarks, positioning them as game-changers in healthcare. However, there remains a persistent gap between high benchmark performance and clinical utility of NLP algorithms due to limitations of existing evaluation paradigms. Existing benchmarks tend to use static, task-specific benchmarks, and as a result, they do not capture the full dimension of complexity, safety, interpretability, and integration in workflow required for the safe deployment in the clinic. The editorial advocates for a shift in evaluation paradigms from narrow score-based metrics to a dynamic, multi-dimensional, and patient-centered system of clinical gatekeeping. The proposed framework integrates a four-phase process, including retrospective benchmarking, pilot testing, multi-center validation, and real-world monitoring, alongside a capability-task-behavior-value progression and a continuous human-in-the-loop feedback mechanism. This comprehensive strategy ensures not only technical robustness but also clinical relevance, ethical accountability, and adaptive improvement, transforming LLMs from experimental tools into reliable clinical partners for safer and more patient-centric healthcare delivery.