Can AI Improve the Readability of Patient Education Information in Gynecology?

Background: The American Medical Association recommends that patient information be written at a 6^th grade level to increase accessibility. However, most existing patient education materials exceed this threshold, posing challenges to patient comprehension. Artificial Intelligence (AI), particularly large language models (LLMs), presents an opportunity to improve the readability of medical information. Despite the growing integration of AI in healthcare, few studies have evaluated the effectiveness of LLMs in generating or improving readability of existing patient education materials within gynecology.

Objective: To assess the readability and effectiveness of patient education materials generated by ChatGPT, Gemini, and CoPilot compared to American College of Obstetricians and Gynecologists (ACOG) and UpToDate.com. Additionally, to determine whether these LLMs can successfully adjust the reading level to a 6^th-grade standard.

Study design: This cross-sectional study analyzed ACOG, UpToDate, and LLM-generated content, evaluating LLMs for two tasks: (1) independent LLM-generated materials and (2) LLM-enhanced versions reducing existing patient information to 6^th-grade level. All materials were assessed for basic textual analysis and readability using eight readability formulas. Two board-certified OBGYNs evaluated blinded patient education materials for accuracy, clarity, and comprehension. ANOVA was used to compare textual analysis and readability scores, with Tukey post-hoc tests identifying differences for both original and enhanced materials. An alpha threshold of p<.004 was used to account for multiple comparisons.

Results: LLM-generated materials were significantly shorter (mean word count 407.9 vs. 1132.0; p<.001) but had a higher proportion of difficult words (36.7% vs. 27.4%; p<.001). ACOG and UpToDate materials averaged 9th and 8.6 grade levels respectively, while AI-generated content reached a 10.6 grade level (p=0.008). Although CoPilot and Gemini improved readability when prompted, no LLM successfully reached the 6th-grade benchmark, and ChatGPT increased reading difficulty.

Conclusions: LLMs generated more concise patient education materials but often introduced more complex vocabulary, ultimately failing to meet recommended health literacy standards. Even when explicitly prompted, no LLM achieved the 6th-grade reading level required for optimal patient comprehension. Without proper oversight, AI-generated patient education materials may create the illusion of simplicity while reducing true accessibility. Future efforts should focus on integrating health literacy safeguards into AI models before clinical implementation.