Comparative benchmarking of the DeepSeek large language model on medical tasks and clinical reasoning

Abstract

DeepSeek is a newly introduced large language model (LLM) designed for enhanced reasoning, but its medical-domain capabilities have not yet been evaluated. This study assessed the capabilities of three LLMs— DeepSeek-R1, ChatGPT-o1, and Llama 3.1-405B—in performing four different medical tasks: answering questions from the United States Medical Licensing Examination (USMLE), interpreting and reasoning based on text-based diagnostic and management cases, providing tumor classification according to RECIST 1.1 criteria, and providing summaries of diagnostic imaging reports across multiple modalities. In the USMLE test, the performance of DeepSeek-R1(accuracy=0.92) was slightly inferior to that of ChatGPT-o1(accuracy=0.95; p = 0.04) but better than that of Llama 3.1-405B (accuracy=0.83; p < 10^-3). For text-based case challenges, DeepSeek-R1 performed similarly to ChatGPT-o1 (accuracy of 0.57 vs 0.55; p = 0.76 and 0.74 vs 0.76; p = 0.06, using New England Journal of Medicine and Medicilline databases, respectively). For RECIST classifications, DeepSeek-R1 also performed similarly to ChatGPT-o1 (0.73 vs 0.81; p = 0.10). Diagnostic reasoning steps provided by DeepSeek were deemed more accurate than those provided by ChatGPT and Llama 3.1-405B (average Likert score of 3.61, 3.22, and 3.13, respectively, p = 0.005 and p < 10⁻³). However, summarized imaging reports provided by DeepSeek-R1 exhibited lower global quality than those provided by ChatGPT-o1 (5-point Likert score: 4.5 vs 4.8; p < 10⁻³). This study highlights the potential of DeepSeek-R1 LLM for medical applications but also underlines areas needing improvements.