Is ChatGPT-4o an Effective Tool for Providing Accurate Responses to Frequently Asked Questions by Varicocele Patients?

Abstract

Background: Varicocele is a common cause of male infertility. Recently, AI tools like ChatGPT have become popular sources of health information among patients.

Aim: We aimed to investigate the effectiveness of ChatGPT-4o in addressing frequently asked questions (FAQs) about varicocele.

Methods: The study included 39 questions, comprising 33 FAQs and 6 questions based on the 2024 guidelines of the European Association of Urology (EAU). The responses provided by ChatGPT-4o were evaluated by two urology specialists for their quality and reliability using the Global Quality Score (GQS). The reproducibility of the responses was analyzed by asking each question twice on different days.

Results: ChatGPT-4o provided high-quality and comprehensive responses (GQS 5) to 30.7% of the questions, good-quality responses (GQS 4) to 59%, and insufficiently comprehensive and moderate-quality responses (GQS 3) to 10.3%. The reproducibility of responses varied, with general questions about varicocele and EAU guideline-based questions showing 83.3% reproducibility, specific questions about varicocele showing 100%, and treatment-related questions showing 76.5%.

Conclusion: This study demonstrates that ChatGPT-4o is a valuable tool in providing information about varicocele, with a high overall success rate in terms of accuracy and reliability. However, it is important to note the lack of excellence in the design of some responses and the low reproducibility of answers, particularly for treatment-related questions.