Comparison of ChatGPT-4, Microsoft Copilot, and Google Gemini for Pediatric Ophthalmology Questions.

Abstract

Purpose: To evaluate the success of Chat Generative Pre-trained Transformer (ChatGPT; OpenAl), Google Gemini (Alphabet, Inc), and Microsoft Copilot (Microsoft Corporation) artificial intelligence (AI) programs, which are offered free of charge by three different manufacturers, in answering questions related to pediatric ophthalmology correctly and to investigate whether they are superior to each other.

Methods: ChatGPT, Gemini, and Copilot were each asked 100 multiple-choice questions from the Ophtho-Questions online question bank, which is widely used for preparing for the high-stakes Ophthalmic Knowledge Evaluation Program examination. Their answers were compared to the official answer keys and categorized as correct or incorrect. The readability of the responses was assessed using the Flesch-Kincaid Grade Level, Flesch Reading Ease Score, and the Coleman-Liau Index.

Results: ChatGPT, Gemini, and Copilot chatbots answered 61 (61%), 60 (60%), and 74 (74%) questions correctly, respectively. The Copilot AI program had a significantly higher rate of correct answers to questions than ChatGPT and Gemini (P = .049 and .035). Three readability analyses revealed that Copilot had the highest average score, followed by ChatGPT and Gemini, which were more challenging than the recommended level.

Conclusions: Although AI chatbots can serve as useful tools for acquiring information on pediatric ophthalmology, their responses should be interpreted with caution due to potential inaccuracies. [J Pediatr Ophthalmol Strabismus. 20XX;X(X):XXX-XXX.].