A comparative analysis of GPT-3.5 and GPT-4.0 on a multiple-choice ophthalmology question bank: A study on artificial intelligence developments.

Abstract

Introduction: To evaluate the performance of ChatGPT-4.0 and ChatGPT-3.5 in answering multiple-choice questions in OphthoQuestions (www.ophthoquestions.com), a popular question preparation bank, and to compare the performance of GPT-4.0 and GPT-3.5.

Methods: In January 2024, using a personal account on OphthoQuestions (www.ophthoquestions.com), 520 questions were selected from 4,551 OphthoQuestions. These 520 questions were created by randomly selecting 40 questions from each of 13 ophthalmology subspecialties. GPT-3.5 and GPT-4.0 were asked to answer these same 520 questions.

Results: ChatGPT-4.0 and ChatGPT-3.5 answered 408 questions (78.46%) 95% CI [70,88%] and 333 questions (64.15%) 95% CI [53,74%] of 520 questions correctly, respectively. GPT-4.0 answered significantly more questions correctly than GPT-3.5 (p= 0.0195). ChatGPT-4.0 showed a statistically significant difference compared to ChatGPT-3.5 in giving correct answers in all subgroup analyses (p<0.05).

Discussions: This study gives an encouraging new proof of ChatGPT's ability to manage complex clinical and medical data, focusing on the development and consistency of artificial intelligence algorithms. The statistically significant success of GPT-4.0 over GPT-3.5 in this study should be examined in light of future algorithm advances, particularly in online tests, which will increase progressively as the use of artificial intelligence poses an increasing danger to test integrity. Protocols such as required proctoring should be considered. In the following years, ChatGPT's clinical management and decision-making expertise should be supplemented by more research indicating that it may be a beneficial resource for ophthalmologists and other medical professionals seeking information and guidance on challenging cases.

Conclusions: GPT-4.0 was found to give more and more consistent answers than GPT 3.5 on a multiple-choice ophthalmology question bank. ChatGPT has shown significant differences between algorithms in accuracy and repeatability when handling questions related to eye diseases. This study shows that new artificial intelligence algorithms are promising. More data is needed to use artificial intelligence language models in medical applications.