MSDiagnosis: A benchmark and framework for evaluating large language models in multi-step clinical diagnosis

Abstract

Clinical diagnosis is critical in clinical decision-making, typically requiring a continuous and evolving process that includes primary, differential, and final diagnoses. However, most existing clinical diagnostic tasks are single-step processes, which do not align with the complex multi-step diagnostic procedures found in real clinical scenarios. In this paper, we propose MSDiagnosis, a Chinese multi-step clinical diagnostic benchmark consisting of 2225 cases from 12 departments, covering primary, differential, and final diagnosis tasks. Conventional approaches often rely on large language models (LLMs) to perform these tasks sequentially, which can lead to error propagation. To address this, we propose a two-stage diagnostic framework consisting of a forward inference module and a backward reasoning and refinement module. This framework is applied at each diagnostic stage to effectively mitigate error propagation across steps. The forward module retrieves similar cases to assist the LLM in generating an initial diagnosis. In the backward inference and refinement module, we first perform backward inference to infer the diagnostic criteria associated with the initially identified potential diseases. These criteria are then compared with the patient’s records to identify and eliminate possible misdiagnoses. Finally, the diagnostic conclusion is further refined and confirmed. Based on the MSDiagnosis, we evaluate medical LLMs (e.g., OpenBioLLM, PULSE, and Apollo2), general LLMs (e.g., DeepSeek-V3, OpenAI-O1, and GLM4), and our proposed framework. Experimental results show that our framework achieves state-of-the-art performance, demonstrating its effectiveness in multi-step diagnostic tasks. We also provide a detailed analysis and suggest future research directions for this task. Our code and data are publicly available at https://github.com/nlper-hou/MSDiagnosis.