Gold-standard-guided direct preference optimisation for ICD-10-AM and ACHI code assignment from discharge summaries.

Abstract

Clinical codes assigned from discharge summaries support clinical research, reimbursement, health resource allocation, and service planning. In Australia, ICD-10-AM and ACHI are used to code diagnoses and interventions for admitted episodes of care, but manual coding is time-consuming, labour-intensive, and subject to variability. This study investigates large language models for automated ICD-10-AM and ACHI code assignment from discharge summaries. We first normalised a public ICD-9-CM coded dataset into the Australian coding system using General Equivalence Mappings and semantic similarity over code descriptions, with validation by a domain expert clinical coder. We then fine-tuned Meta-Llama-3-8B and compared full-parameter fine-tuning with a parameter-efficient QLoRA baseline. Finally, we applied Direct Preference Optimisation (DPO) using gold-standard-guided preference pairs, where the gold-standard ICD-10-AM/ACHI code list was treated as the preferred response and the supervised fine-tuned model output as the non-preferred response. Experiments were conducted on a mapped MIMIC-III subset and a de-identified Australian hospital dataset. Among the evaluated model variants, the DPO-aligned full-parameter model achieved the highest score, with micro-F1 scores of 0.7397 on the mapped dataset and 0.7834 on the Australian hospital dataset. These findings suggest that gold-standard-guided DPO can improve agreement with reference code sets compared with supervised fine-tuning alone. However, the results are limited to high-frequency diagnosis and procedure codes, short-document evaluation settings, and DPO preference pairs reviewed by one annotator.