LLM4DEU: Fine Tuning Large Language Model for Medical Diagnosis in Outpatient and Emergency Department Visits of Neurosurgery

Abstract

Clinical diagnosis for complex disease conditions is a complicated decision process involving systematic inference and differentiation. Artificial Intelligence (AI) models have been a widely established approach to help improve the efficiency of various kinds of clinical decision tasks (e.g., diagnosis, treatment, and prognosis). However, due to the critical requirement of time efficiency, lack of sufficient information, and high probability of comorbid diseases in Outpatient and Emergency Settings (OES), it is still challenging to build clinically feasible AI models using the free text clinical records in OES for complex disease conditions, such as neurosurgery. Here we propose an AI diagnosis model, named LLM4DEU, for neurosurgery disease differentiations by fine-tuning a large language model (i.e., ChatGLM) using the Department of Neurosurgery, the Beijing Tiantan Hospital OES electronic health records. LLM4DEU obtained state-of-the-art performance on clinical diagnosis with a F1 score of 78.53%, which is superior to five well-known baselines (including deep learning models). In addition, we evaluated the actual performance of the model by case studies on the diagnosis of specific neurosurgical diseases (e.g., subdural hematoma, cerebral hemorrhage, and cerebral infarction). The experimental results show that the LLM4DEU model has significant advantages in diagnosing low-incidence disease conditions, and comparative analyses with clinical experts confirm the predictive power of the model in neurosurgical diagnosis.