Can AI chatbots accurately answer patient questions regarding vasectomies?

Abstract

Artificial Intelligence (AI) has revolutionized the healthcare industry. There have been limited studies assessing AI model efficacy and accuracy in urology. To our knowledge, there is a lack in research looking at one of the most common urological procedures: the vasectomy. Ten frequently asked questions regarding vasectomies were individually entered into three different AI sources (ChatGPT, Bard & Bing) using free interfaces available to consumers. The responses were critically analyzed by three urologists and graded on a scale of 1 to 4 for clarity, accuracy, and evidence-based information, with 1 being the best and 4 being the worst. ChatGPT had the best average rating per question at 1.367, followed by Bard at 2.167 and Bing at 1.800(p = 0.000083). ChatGPT was found to provide significantly more satisfactory answers than both Bard (p = 0.00005) and Bing (p = 0.03988). The difference between Bard and Bing however was found to be insignificant (p = 0.09651). Overall, our study shows that AI Chatbots may provide mostly accurate information on frequently asked questions regarding vasectomies and is a reasonable resource for patients interested in the procedure to use. ChatGPT is the most accurate and concise of the chatbots assessed.