Evaluating large language models for glaucoma counseling: A pilot study of ChatGPT and Google Gemini responses in Traditional Chinese.

Abstract

Purpose: Large language models (LLMs) such as ChatGPT and Google Gemini are increasingly used for patient education in various medical fields. However, their effectiveness in providing ophthalmology-related information in non-English languages remains underexplored. This study aimed to evaluate and compare the appropriateness of ChatGPT and Google Gemini responses to glaucoma-related questions presented in Traditional Chinese.

Materials and methods: Twenty frequently asked glaucoma-related questions were translated into Traditional Chinese and categorized into four domains: disease understanding, diagnosis, treatment, and lifestyle. Responses were generated using ChatGPT (GPT-4) and Google Gemini. Three qualified ophthalmologists independently rated the appropriateness of each response using a 5-point Likert scale. Mean scores were calculated, and statistical comparisons were performed for overall and category-specific performance.

Results: ChatGPT achieved a higher overall appropriateness score (4.32 ± 0.65) than Google Gemini (4.03 ± 0.66), with a statistically significant difference (P = 0.004). Across all four content domains, ChatGPT consistently outperformed Gemini, although the differences did not reach statistical significance. Both models demonstrated relatively lower performance in lifestyle-related questions.

Conclusion: Both ChatGPT and Google Gemini are capable of providing reasonably appropriate glaucoma-related information in Traditional Chinese. ChatGPT demonstrated superior overall performance. These findings support the potential role of LLMs in enhancing patient education in non-English settings, although further improvements in language localization and domain-specific accuracy are warranted.