Identification of Syndrome Types in Patients With Pancreatic Cancer From Free Text in Electronic Medical Records: Model Development and Validation.

Abstract

Background: Syndrome differentiation is crucial in traditional Chinese medicine (TCM) diagnosis and treatment, but it heavily relies on expert experience, limiting systematic standardization.

Objective: This study developed and validated a BERT (bidirectional encoder representations from transformers)-based model, the traditional Chinese medicine pancreatic cancer syndrome differentiation bidirectional encoder representations from transformers (TCMPCSD-BERT), using in-house pancreatic cancer medical records, to digitalize expert knowledge and support standardized syndrome differentiation in TCM.

Methods: A retrospective dataset of pancreatic cancer cases (2011-2024) from Fudan University Shanghai Cancer Center was annotated into 4 TCM syndrome types by 2 experts (Cohen κ=0.913). The proposed TCMPCSD-BERT model was compared with conventional models (long short-term memory and text convolutional neural network) embedded in TCM diagnostic tools and with large language models (LLMs; ChatGPT-4o, Kimi, Ernie Bot 4.0 Turbo, and Zhipu Qingyan) under a prompt engineering framework. Performance evaluation on in-house data was supplemented with attention visualizations and integrated gradients analyses for interpretability. The McNemar test assessed classification accuracy differences, while bootstrap 95% CIs quantified statistical uncertainty and stability. The Welch t test (2-tailed) was used to evaluate mean differences between TCMPCSD-BERT and the comparator models.

Results: Among 6830 records, case counts were damp-heat syndrome (n=1694), spleen-deficiency syndrome (n=1185), damp-heat with spleen-deficiency syndrome (n=1178), and others (n=2773). On the test set, McNemar test showed significantly higher accuracy for TCMPCSD-BERT than the 3 baseline models and generally better performance than LLMs. In all comparisons, TCMPCSD-BERT achieved higher mean macroprecision, macrorecall, macro-F₁-score, and accuracy, with nonoverlapping 95% bootstrap CIs and significant Welch t test results (P<.01). The model achieved a macroprecision of 0.935 (95% CI 0.918-0.951), macrorecall of 0.921 (95% CI 0.900-0.942), macro-F₁-score of 0.927 (95% CI 0.908-0.945), and accuracy of 0.919 (95% CI 0.899-0.939). Attention visualizations suggested the model could capture less common TCM term associations, while integrated gradients highlighted high-attribution diagnostic features (eg, "gray-white stool" 0.933 in damp-heat syndrome; "indigestion" 1.204 in spleen-deficiency syndrome). Misclassification analyses indicated challenges in handling overlapping or atypical symptom presentations. Compared with LLMs, web-based platforms, and diagnostic instruments, TCMPCSD-BERT appeared to provide relatively higher accuracy, interpretability, and efficiency in processing long unstructured texts for syndrome differentiation.

Conclusions: The TCMPCSD-BERT model shows potential for automated syndrome differentiation from unstructured clinical texts and broader application in TCM. Based on this study, it appears to improve operability over 4-diagnostic instruments and web-based platforms, and offers greater stability and accuracy than LLMs in specific tasks. However, these findings should be interpreted cautiously, given the subjectivity of syndrome definitions, data imbalance, and reliance on preprocessed, expert-annotated data. Further studies involving larger and more diverse populations are needed to validate its generalizability and support its broader application in real-world settings.