Quality of information on hypospadias from artificial intelligence chatbots: How safe is AI for patient and family information?

Abstract

Introduction: Hypospadias is the most prevalent congenital anomaly of the penis, with an estimated incidence of 0.4-8.2 cases per 1000 live births (1). However, most of the parents and families of those with hypospadias experience anxiety and uncertainty regarding the information about hypospadias (2, 3). Leading to many families conduct their own independent internet search for information to better understand a diagnosis. The reliability and quality of this information for patients and families has not previously been formally assessed. The objective of this study is to assess the ability of AI chatbots to provide accurate and readable information to patients and families on hypospadias.

Methods: AI chatbot inputs were sourced from google trends and healthcare organisations. Google trends was used to identify the top 10 google search terms relating to 'Hypospadias' based on search volume. Royal Children Hospital in Melbourne (RCH) and the Urology Care Foundation American Urology Association - Hypospadias (AUA) headers were used as healthcare related hypospadias inputs4 different AI chatbot programs ChatGPT version 4.0, Perplexity, Chat Sonic, and Bing AI. Three urology consultants blinded to the AI chatbots assessed responses for accuracy and safety and a further two trained investigators, blinded to AI chatbot type and each other's evaluation scores, assessed AI chatbot responses using various evaluation instruments including PEMAT, DISCERN, misinfomration and Flesch-Kincaid readability formula as well as word count and citation.

Results: As demonstrated in the 4 AI chatbots assessed contained high quality health consumer information median DISCERN 4 (IQR 3-5). The degree of misinformation was low overall and across all AI chatbot responses, with a median of 1 (IQR 1-1). The PEMAT Understandability scores was high overall with a median of 91.7 % (IQR 80-92.3). However, all AIs performed poorly in the actionability of their responses with an overall median of 40 % (20-80). The median word count per AI chatbot response was 213 (IQR 141-273).

Conclusion: AI chatbots provided understandable, high level and accurate health information relating to hypospadias. However, the information was delivered at a reading level which may limit its use in a paediatric or general public setting, and only one chatbot gave clearly actionable interventions or direction. Overall, AI chatbots are a clinically safe and appropriate adjunct to face to face consultation for healthcare information delivery and will likely take a more prominent domain as technology advances.