Filling the information gap: a validated comparison of ChatGPT, Gemini, and Claude for penile cancer patient education.

Abstract

Background: ChatGPT, Gemini, and Claude are three large language models (LLMs) increasingly consulted by patients researching a new cancer diagnosis. Penile cancer is rare, stigmatised, and dominated by decisions around organ preservation and sexual function, which makes the quality of patient-facing information particularly consequential. The aim of this study was to assess the quality of responses generated by these LLMs to patient-oriented questions on penile cancer.

Methods: Fourteen questions reflecting common patient information needs were adapted from established educational resources and refined by senior urology consultant review. Each query was presented to ChatGPT, Gemini, and Claude using an identical patient-centred prompt. Responses (n=42) were anonymised and randomly ordered before being independently reviewed in a consensus meeting by a urology panel using the Quality Analysis of Medical Artificial Intelligence (QAMAI) framework. Comparative analysis used the Friedman test with Bonferroni-corrected pairwise Wilcoxon signed-rank testing.

Results: Mean total QAMAI scores were 22.7 for ChatGPT (median, 23; range, 17 to 25), 22.6 for Claude (median, 23; range, 18 to 24), and 17.0 for Gemini (median, 17.5; range, 14 to 20), corresponding to moderate quality for ChatGPT and Claude and the boundary of poor and moderate quality for Gemini. The difference across models was significant (Friedman χ²=18.78; P<0.001). ChatGPT and Claude each significantly outperformed Gemini (both adjusted P=0.004) but did not differ from one another (adjusted P>0.99). ChatGPT produced the clearest responses and Claude the most complete, while Gemini scored lowest in every domain, most strikingly for surgery and organ preservation (14.0 vs. 23.7 and 23.0). No response provided citations or referred readers to external sources of information.

Conclusions: ChatGPT and Claude delivered broadly equivalent, moderate-to-good-quality information on patient-derived penile cancer questions, and both clearly outperformed Gemini, which frequently produced incomplete answers in the surgical and organ preservation domain. The universal absence of source citation is a clinically relevant limitation. These tools should be regarded as a supplement to, rather than a substitute for, specialist consultation.