A Comparative Assessment of Large Language Models in Pediatric Dialysis: Reliability, Quality and Readability.

Abstract

Introduction: This study evaluated the reliability, quality, and readability of ChatGPT (OpenAI, San Francisco, CA), Gemini (Google, Mountain View, CA), and Copilot (Microsoft Corp., Washington, DC) which are among the most widely used large language models (LLMs) today in answering frequently asked questions (FAQs) related to pediatric dialysis.

Methods: A total of 45 FAQs were entered into LLM. The Modified DISCERN (mDISCERN) scale assessed reliability; the Global Quality Score (GQS) evaluated quality; and readability was assessed using five metrics: Coleman-Liau Index (CLI), Simple Measure of Gobbledygook (SMOG), Gunning Fog Index (GFI), Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL). Questions were directed to the chat robots twice, on January 25, 2025, and February 1, 2025.

Results: All three chatbots displayed high reliability, achieving median mDISCERN scores of 5. Quality scores on the GQS were similarly high, with median scores of 5 across platforms; however, Gemini exhibited greater variability (range 1-5) compared to ChatGPT-4o and Copilot (ranges 3-5). Readability scores revealed that chatbot responses were written at an advanced level.

Conclusion: This study found that LLMs responses to dialysis FAQs were reliable and high quality, but difficult to read; improving readability through expert-reviewed content could increase their impact on public health.