Performance of the large language model in general medicine

Abstract

Background

Domain-specific large language models (LLMs) may aid primary care, but reliability and safety are uncertain, and general-practice evaluations remain limited.

Methods

Observational, exam-based comparison (Feb–May 2025) using standardized items from the Chinese General Practitioner Licensing Examination. A general-practice LLM (AIGP) was evaluated against 389 physicians on the same items under deterministic decoding. Each item was run 50 technical replicates. Outcomes: accuracy; stability (coefficient of variation, ICC(2,k)); and expert-rated potential clinical harm for incorrect outputs.

Results

AIGP achieved 88.41 % ± 1.87 accuracy vs physicians 72.12 % ± 4.97 (mean difference 16.29, 95 % CI 15.56–17.01; P < .001). Stability favored AIGP (CV 2.12 vs 6.89; agreement across replicates ICC(2,k) = 0.993, 95 % CI 0.991–0.995). Although overall errors were fewer, AIGP's mistakes were systematic and carried higher average harm ratings, clustering in red-flag contexts (pregnancy, pediatrics, anticoagulation, acute chest pain).

Conclusions

In a controlled test setting, AIGP outperformed physicians and showed excellent stability. Findings are exploratory and context-limited (exam-based, text-only) and reveal systematic higher-harm failure modes. AIGP is a starting point for decision support, not a deployable clinical tool without prospective real-world validation and safety guardrails (human-in-the-loop review, contraindication/dose checks, uncertainty-aware refusal).