Artificial Intelligence Performance Under Different Conditions in Answering China's Standardized Training Examination for Resident Physician in Radiology: A Comparative Analysis

Abstract

Background

The capabilities of general-purpose large language models (LLMs) on specialized medical examinations have not been systematically compared. To evaluate the performance differences among three LLMs—DeepSeek-R1, ChatGPT-o1, and Gemini-2.0—in answering questions from China's Standardized Training Examination for Resident Physicians in Radiology, and to assess the impact of different questioning conditions (with/without answer choices, and the introduction of doubt) on model accuracy.

Methods

A total of 131 questions were analyzed at one time. The LLMs were subjected to two tasks (questions with/without answer choices). Each task included three conditions: no doubt, weak doubt, and strong doubt, with the latter two presented as follow-up questions after the models' initial responses. Subjective evaluation was conducted using the 5-point Likert scale.

Results

In both tasks, Gemini-2.0 achieved the highest accuracy (0.763–0.809) and (0.595–0.679). In Task 2, all LLMs' accuracy was lower than in Task 1, but only DeepSeek-R1 showed statistical significance (p = 0.014). The introduction of doubt (weak or strong) did not significantly increase accuracy in Task 1 for any model. In Task 2, only DeepSeek-R1 showed a slight increase in accuracy under doubt conditions. All LLMs performed significantly better on single-choice questions than on multiple-choice questions (p < 0.05) and showed superior performance in case-based questions vs. knowledge-based questions. The subjective evaluation scores were 4.46 for DeepSeek-R1, 3.92 for ChatGPT-o1, and 3.83 for Gemini-2.0. It was noted that ChatGPT-o1 negotiated not to answer all questions at once and had missing responses.

Conclusions

LLMs exhibit variable proficiency in tackling radiology resident examination questions, with Gemini-2.0 showing the highest overall accuracy. However, repeated self-examination of LLMs through the introduction of doubt does not consistently or significantly enhance their performance on radiology-related questions.