Automating Injury Severity Score Calculation From Free-Text Discharge Diagnoses Using Domain-Specific Retrieval-Augmented Large Language Models: Retrospective Validation Study.

Abstract

Background: Trauma registries are crucial for evaluating the prognosis of injuries and the quality of trauma care. However, the calculation of Injury Severity Score (ISS) remains a predominantly manual process, which is labor-intensive, potentially error-prone, and difficult to scale. Automating this process is a critical step toward enhancing the efficiency and quality of trauma registry data.

Objective: This study conducts a feasibility test on the use of large language models (LLMs) to automatically calculate ISS from free-text discharge diagnoses. We specifically investigate the efficacy of retrieval-augmented generation (RAG) to enhance LLM accuracy by incorporating knowledge from the Abbreviated Injury Scale. We compare the performance of baseline LLMs against models augmented with both a standard RAG and a novel, domain-specific RAG framework.

Methods: In this retrospective validation study, we used deidentified, free-text discharge diagnoses from trauma patients at Chang Gung Memorial Hospital (Linkou, Taiwan) in 2022. Against a physician-verified gold standard, we compared the performance of three systems: (1) baseline LLMs, (2) LLMs with standard RAG, and (3) LLMs with domain-specific RAG, for which the retriever was fine-tuned on trauma-related ICD-10 (International Statistical Classification of Diseases, Tenth Revision) codes. Performance was evaluated on a physician-verified test set across exact ISS prediction, tolerance-based accuracy, and ISS severity categories.

Results: Baseline state-of-the-art LLMs achieve modest exact-match (EM) accuracy, ranging from 47.3% to 64.7%. Standard RAG did not produce consistent gains (54.3% to 60.3%), whereas the domain-specific RAG substantially improved accuracy (62.7% to 75.3%). Performance gains were most pronounced in patients with moderate to severe injuries. GPT-4.1 with domain-specific RAG reached the highest EM accuracy (75.3%, 95% CI 66.1%-82.7%), compared with 70.0% for the deterministic International Classification of Diseases and R statistical software (ICDPIC-R) baseline. Tolerance-based and categorical analyses confirm substantial gains with the domain-specific RAG framework in clinically relevant ranges.

Conclusions: This single-center feasibility study suggests that LLMs augmented with a domain-specific retriever can calculate the ISS from free-text discharge diagnoses with overall accuracy comparable to that of a deterministic coding tool, with the advantage concentrated in the most severe injuries. This approach may help alleviate the administrative burden of manual coding, enhance standardization, and improve the quality of trauma registry data. Future multi-institutional validation is needed to confirm these findings.