Assessment of artificial intelligence performance in answering questions on onabotulinum toxin and sacral neuromodulation.

Abstract

Purpose: This study aimed to evaluate the performance of three artificial intelligence (AI) models-ChatGPT, Gemini, and Copilot-in addressing clinically relevant questions about onabotulinum toxin and sacral neuromodulation (SNM) for the management of overactive bladder (OAB).

Materials and methods: A set of 30 questions covering mechanisms of action, indications, contraindications, procedural details, efficacy, and safety profiles was posed to each AI model. Responses were assessed by a panel of four urology specialists using predefined criteria: accuracy, completeness, clarity, and consistency. A multi-dimensional scoring framework evaluated the performance across five dimensions: factual accuracy, relevance, clarity/coherence, structure, and utility. Responses were scored on a 4-point Likert scale, and statistical analyses were conducted using one-way ANOVA to compare model performance.

Results: ChatGPT achieved the highest mean score (3.98/4) across all dimensions, with statistically significant differences compared to Gemini (3.20/4) and Copilot (2.60/4) (p=0.001 for all dimensions). ChatGPT excelled particularly in clinical application, procedure, and safety categories, consistently delivering accurate and comprehensive answers. No statistically significant differences were found between Gemini and Copilot in most categories.

Conclusions: ChatGPT demonstrated superior performance in generating accurate, complete, and clinically relevant responses for OAB management, highlighting its potential as a reliable tool for both healthcare professionals and patients. However, the variability observed in Gemini and Copilot underscores the need for further refinement of these models. Future studies should explore real-world integration of AI models into clinical workflows to enhance patient care and decision-making.