Reliability, Quality, and Readability of Freely Available Large Language Models for Clinical Decision Support in Perforated Peptic Ulcer: A Blinded, Multidimensional Study.

Abstract

Background/Objectives: Perforated peptic ulcer (PPU) is a common surgical emergency in which each hour before surgery increases mortality, and the decision is often made by a single surgeon on call. Large language models (LLMs) may offer rapid decision support, but their reliability in this setting is unknown. We compared five freely accessible LLMs in a blinded, multidimensional design. Methods: Twenty-four structured questions derived from the WSES 2020 guideline, covering the preoperative, intraoperative, and postoperative phases, were posed to the free versions of ChatGPT (GPT-5.3 Instant), Claude (Sonnet 4.6), DeepSeek-V3, Gemini (2.0 Flash), and Grok 3 during April and May 2026, yielding 120 responses. Two general surgeons blinded to model identity scored scientific reliability with the modified DISCERN instrument (mDISCERN, 8-40) as the primary measure and overall quality with the Global Quality Score (GQS, 1-5) as a secondary measure. Because every question was answered by all five models, the Friedman test was primary. Results: Models differed significantly in reliability (Friedman chi-square(4) = 71.35, p < 0.001, Kendall W = 0.743) and separated into two tiers: Claude (mDISCERN 37.83), Grok 3 (36.08), and DeepSeek-V3 (34.92) scored higher than ChatGPT (30.62) and Gemini (29.48). All between-tier comparisons were significant, whereas no within-tier comparison reached significance. Inter-rater agreement was moderate (mDISCERN ICC(2,k) = 0.647) and internal consistency high (Cronbach alpha 0.867 and 0.916). Reliability correlated positively with response length (rho = 0.717) and negatively with readability (rho = -0.452). Conclusions: Freely accessible LLMs vary substantially in reliability for PPU, so model choice has clinically meaningful consequences. This variation is specific to the models, versions, and access tier evaluated, and no model should be used for emergency surgical decision support without independent, setting-specific validation.