Evaluation of the large language models Chatbot's responses to frequently asked queries on refractive errors.

Abstract

PurposeTo assess the accuracy, comprehensiveness, and reliability of Large Language Model chatbots in answering frequently asked questions about refractive errors.MethodsForty-four questions about refractive errors were posed to four chatbots, including Copilot, Perplexity, Gemini, and ChatGPT. Responses to each question were independently evaluated by three experts using a three-point accuracy scale. The readability of the chatbots' responses was evaluated using several indices. Similarity was assessed using Sentence-Bidirectional Encoder Representations from Transformers (SBERT). Inter-rater agreement among the graders was evaluated using the Gwet Agreement Coefficient 1 (AC1) statistic.ResultsThe overall agreement among the graders for all chatbot responses was almost perfect (Gwet AC1: 0.87). All chatbots received scores above 85% in every category, and there was no significant difference in chatbot accuracy scores (P= 0.168). The highest mean comprehensiveness score was observed for Perplexity (8.13 ± 0.95, P = 0.028). The similarity scores of the chatbots were very close to each other. All readability scores showed significant differences (P<0.001) between the chatbots.ConclusionAll four chatbots showed comparable accuracy in answering questions about refractive errors. Readability levels of all chatbots exceeded public health recommended thresholds; however, ChatGPT produced relatively more accessible output.