Testing the Reliability of ChatGPT Assistance for Surgical Choices in Challenging Glaucoma Cases.

Abstract

Background: This study's aim is to assess ChatGPT's capability to analyze detailed case descriptions in glaucomatous patients and suggest the best possible surgical treatment. Methods: We conducted a retrospective analysis of 60 medical records of surgical glaucoma cases, divided into "ordinary" cases (n = 40) and "challenging" cases (n = 20). We entered every case description into ChatGPT-3.5's interface and inquired "What kind of surgery would you perform?". The frequency of accurate surgical choices made by ChatGPT, compared to those reported in patients' files, was reported. Furthermore, we assessed the level of agreement with three senior glaucoma surgeons, asked to analyze the same 60 cases and outline their surgical choices. Results: Overall, ChatGPT surgical choices were consistent with those reported in patients' files in 47/60 cases (78%). When comparing ChatGPT choices with the three glaucoma specialists, levels of agreement were 75%, 70%, and 83%, respectively. In ordinary cases, we did not report any significant differences when comparing ChatGPT answers with those of the three glaucoma specialists, when both of them were matched with patients' files (p > 0.05 for all). ChatGPT's performances were lower in "challenging" cases: when compared to patients' files, the accuracy was 13/20 (65%); when compared to glaucoma specialists, the level of agreement was 50%, 40%, and 70%, respectively. Conclusion: In ordinary conditions, ChatGPT was able to propose coherent personalized treatment plans, and its performance was comparable to that of skilled glaucoma specialists but showed its limitations in the evaluation of more complex cases.