Comparison of DeepSeek AI and Human Reviewers in Structured Evaluation of Neurosurgical Manuscripts: A Pilot Study.

Abstract

Objectives Increasing manuscript volume has strained peer review, prompting interest in artificial intelligence (AI) as a supportive tool. We evaluated DeepSeek R1 as a structured peer reviewer of neurosurgical case-based manuscripts and compared its scoring patterns with those of an experienced editor and student reviewers. Design Exploratory, cross-sectional comparative study using a convenience sample of consecutively submitted manuscripts. Setting Single skull-base neurosurgical journal during a 6-month submission window (January-June 2025). Participants Three reviewer groups independently assessed 11 de-identified, unpublished neurosurgical manuscripts: first-year medical students ( n = 154; each randomly assigned one manuscript after a course in biostatistics, epidemiology, and critical appraisal), the journal's Editor-in-Chief (Experienced Reviewer), and DeepSeek R1. Main Outcome Measures Each manuscript was rated on a 3-point Likert scale (1 = low, 2 = intermediate, 3 = high) across five domains: Study Objective, Novelty, Study Design, Thoroughness, and Clarity. Results Significant between-group differences were found in four of five domains: Study Objective ( H = 8.31, p = 0.02), Novelty ( H = 9.74, p = 0.008), Thoroughness ( H = 10.49, p = 0.005), and Clarity ( H = 8.13, p = 0.02); Study Design did not differ ( H = 1.45, p = 0.483). DeepSeek scored significantly higher than the Experienced Reviewer for Study Objective ( p = 0.015) and higher than both human groups for Novelty. Conclusions DeepSeek showed consistency in structured domains but systematically inflated subjective ratings, particularly Novelty. A human-in-the-loop model integrating AI efficiency with expert judgment may optimize peer review.