Evaluation of DeepSeek with a RAG-based knowledge base in preoperative assessment for arthroplasty to improve the decision-making accuracy of junior physicians.

Abstract

Background: Patients undergoing elective joint replacement are mostly elderly individuals with multiple chronic diseases. These patients have a higher risk, so preoperative assessment for joint replacement is vital for surgical safety but remains challenging for junior physicians. This study aimed to evaluate the decision-making performance of different versions of DeepSeek (Chat and R1) and their knowledge base (KB)-enhanced schemes in perioperative risk assessment and their impact on junior physicians' decision-making quality.

Methods: A total of 292 anonymized medical records from joint replacement patients were retrospectively analyzed. A professional knowledge base was constructed using 21 perioperative management guidelines and retrieval-augmented generation (RAG) technology. The performance of four models (DeepSeek-Chat, DeepSeek-Chat + KB, DeepSeek-R1, and DeepSeek-R1 + KB) was compared in ASA scoring, RCRI scoring, surgical tolerance judgment, and risk prediction. These indicators are related to specific decisions in joint replacement surgery, including preoperative optimization before elective TKA/THA, mitigation of infection risk, planning of VTE prophylaxis, and indication for postponing elective arthroplasty, etc. Senior physicians evaluated model performance using a Likert scale. A 5 × 5 balanced Latin square crossover design was used to assess the impact of model assistance on junior physicians' clinical decision-making.

Results: No significant difference was found in ASA scoring accuracy (p > 0.05). For RCRI scoring and risk stratification, DeepSeek-R1 models (79.1%-79.7%) outperformed Chat models (56.5%-66.0%, p < 0.05). In surgical tolerance assessment, DeepSeek-R1 + KB showed relatively better performance, with significantly better predictive precision (p = 0.0284) and a lower missed diagnosis rate (p = 0.005) compared to the Chat models. The R1 series also showed higher predictive efficiency (87.2%-86.4% vs. 63.6%-61.8%, p < 0.05). The R1 models showed better performance than Chat models in medical comprehension, retrieval accuracy, and reasoning logic. After adjusting for physician variability and patient case mix, the DeepSeek-R1 + KB showed a statistically significant improvement over independent decision-making (OR = 1.58, p = 0.004), as well as the DeepSeek-R1 and Chat series.

Conclusion: The DeepSeek-R1 model, with chain-of-thought reasoning, shows a marginal advantage over the Chat version in preoperative assessment for joint replacement. As a cost-effective clinical auxiliary tool, its integration with a professional knowledge base scheme can optimize the decision-making quality of junior physicians, particularly in terms of providing helpful and comprehensive decision support and providing novel insights for individualized assessment in the context of precision medicine.