Evaluating large language models as an educational tool for meningioma patients: patient and clinician perspectives.

Abstract

Background: The study explores the potential of ChatGPT, an advanced large language model (LLM) by OpenAI, in educating patients about meningioma, a common type of brain tumor. While ChatGPT has generated significant debate regarding its utility and ethics, its growing popularity suggests that patients may increasingly use such tools for medical information. The study specifically examines how patients who have undergone radiation therapy for meningioma perceive the information generated by ChatGPT, integrating both patient feedback and clinical assessment.

Methods: Eight meningioma-related questions on diagnosis, treatment options, and radiation therapy were posed to ChatGPT 4. A questionnaire with these responses and feedback items was developed to assess utility, accuracy, clarity, and alignment with patients' experiences. Nine clinicians first rated each response's relevance, correctness, and completeness on a five-point Likert scale. Subsequently, 28 patients with meningioma completed the questionnaire during their first follow-up visit (three months post-radiation therapy). Finally, the same questions were presented to three other large language models (ChatGPT 4o mini, Gemini Free, Gemini Advanced), and seven blinded clinicians rated each model's responses before selecting the most accurate, eloquent, and comprehensive overall.

Results: The study cohort included 28 meningioma patients, mostly female, with a median age of 60 years. Most patients found the information clear, accurate, and reflective of their experiences, with 60% willing to use ChatGPT for future inquiries. Clinicians rated the relevance and correctness of the information highly, although completeness was rated slightly lower, particularly for questions about specific radiation therapy details and side effects. ChatGPT 4 and its newer version ChatGPT 4o mini received the highest, nearly identical scores among the four LLMs evaluated, while Gemini Free scored the lowest in clinician assessments.

Conclusions: ChatGPT demonstrates potential as a supplementary educational tool for meningioma patients, though some areas may require improvement, particularly in providing comprehensive information. The study highlights the potential for integrating AI in patient education, while also noting the need for clinical oversight to ensure accuracy and completeness.

Trial registration: LMU ethics vote nr.: 23-0742.