Enhancing diabetic retinopathy query responses: assessing large language model in ophthalmology

Abstract

Background Diabetic retinopathy (DR) is a leading cause of blindness, with an increasing reliance on large language models (LLMs) for health-related information. The specificity of LLM-generated responses to DR queries is yet to be established, prompting an investigation into their suitability for ophthalmological contexts. Methods A cross-sectional study involving six LLMs was conducted to ascertain the accuracy and comprehensiveness of responses to 42 DR-related questions from 1 February 2024 to 31 March 2024. Three consultant-level ophthalmologists independently assessed the responses, grading them on accuracy and comprehensiveness. Additionally, the self-correction capability and readability of the responses were analysed statistically. Results An analysis of 252 responses from six LLMs showed an average word count ranging from 155.3 to 304.3 and an average character count ranging from 975.3 to 2043.5. The readability scores showed significant variability, with ChatGPT-3.5 displaying the lowest readability level. The accuracy of the responses was high, with ChatGPT-4.0 receiving 97.6% good ratings and no ‘poor’ grades for the top three models. After introducing a self-correction prompt, the average accuracy score demonstrated a significant improvement, increasing from 6.4 to 7.5. Conclusion LLMs have the potential to provide accurate and comprehensive responses to DR-related questions, making them advantageous for ophthalmology applications. However, before clinical integration, further refinement is needed to address readability, and continuous validation assessments are imperative to ensure reliability. Data are available upon reasonable request.