Artificial Intelligence Outperforms Physicians in General Medical Knowledge, Except in the Paediatrics Domain: A Cross-Sectional Study.

Abstract

Generative artificial intelligence (genAI) shows promising results in clinical practice. This study compared a GPT-4-turbo virtual assistant with physicians from Italy, France, Spain, and Portugal on medical knowledge derived from national exams while analysing knowledge retention over time and domain-specific performance. Via a digital platform, 17,144 physicians provided 221,574 answers to 600 exam questions between December 2022 and February 2024. Physicians were stratified by years since graduation and specialty, and the assistant answered the same questions in each native language. Differences in proportions of correct answers were tested with binomial logistic regression (odds ratios, 95% CI) or Fisher's exact test (α = 0.05). The assistant outperformed physicians in all countries (72-96% vs. 46-62%; logistic regression, p < 0.001). Physicians also trailed the assistant across most knowledge domains (p < 0.001), except paediatrics (45% vs. 52%; Fisher, p = 0.60). Accuracy declined with seniority, falling 4-10% between the youngest and oldest cohorts (logistic regression, p < 0.001). Overall, genAI exceeds practising doctors on broad medical knowledge and may help counter knowledge attrition, though paediatrics remains a domain requiring targeted refinement.