Dr. LLM Will See You Now: The Ability of ChatGPT to Provide Geographically Tailored Colorectal Cancer Screening and Surveillance Recommendations.

Abstract

Background/Objectives: This study evaluates the performance of a large language model (lLm) in providing geographically tailored colorectal cancer screening and surveillance recommendations to gastrointestinal surgeons. Methods: Fifty-four patient cases, varying by age and family history, were developed based on colorectal cancer guidelines. Standardized prompts with predefined query terms were used to query ChatGPT-4.5 on 18 April 2025, from four locations: Canada, Italy, Romania, and the United Kingdom. Responses were classified as "Correct," "Partially Correct," or "Incorrect" based on clinical guidelines and expert recommendations for each country. Outcomes were analyzed using descriptive statistics. Results: ChatGPT provided recommendations on screening eligibility, test interpretation, the management of positive results, and surveillance intervals. Correct recommendations were given for 50.0% (27/54) of cases in Canada, 63.0% (34/54) of cases in Italy, 40.7% (22/54) of cases in Romania, and 55.6% (30/54) of cases in the United Kingdom. Queries in Italian yielded correct guidance for 64.8% (35/54) of cases, while Romanian queries were accurate for 40.7% (22/54) of cases. Notably, Romania and Italy lacked detailed guidelines for polyp management and post-test surveillance. A key finding was the inconsistency between ChatGPT-generated titles and corresponding recommendations, which may impact its reliability in clinical decision-making. Conclusions: ChatGPT-4.5's performance varies by country and language, highlighting inconsistencies in geographically tailored recommendations. This study highlights limitations associated with the training data cutoff and the potential biases introduced by model-generated responses. Healthcare professionals should recognize these limitations and the possible gaps in guideline availability, particularly for high-risk screening, polyp management, and surveillance in certain European countries.