Characterising liver lesions from free-text computer tomography reports – A real-world multicentre analysis

Abstract

Background

This study evaluates the performance of a general-purpose (GPT-4) and a medically fine-tuned (Med-LM) large language model (LLM) in classifying liver lesions from unstructured Computed Tomography (CT) reports.

Methods

Consecutive CT reports (2014–2020) from five institutions were input into GPT-4 and Med-LM with simple (sp) and optimised (op) prompts. Lesion- and patient-level performance were benchmarked against LI-RADS scores assigned by two radiologists, and report quality was analysed using a 5-point Likert scale.

Results

A total of 296 CT reports (mean age, 64.6 years ± 11.3 [SD]; 193 men; 654 lesions) were included. Lesion- and patient-level accuracies for LI-RADS scoring ranged from 40.8% (Med-LMsp) to 61.3% (Med-LMop) and from 27.7% (Med-LMsp) to 52.4% (Med-LMop), respectively. When dichotomized into malignant and benign lesions, lesion- and patient-level accuracies rose to 56.1% (GPT-4sp) − 82.3% (Med-LMop) and 71.3% (Med-LMsp) – 86.5% (Med-LMop). Med-LMop demonstrated the highest performance in all analyses and was statistically superior to other models (all p < 0.001). Non-classification rates ranged between 12.7% (Med-LMop) and 40.5% (GPT-4sp), particularly for benign lesions. Kappa values were weak to moderate between the two reviewers in different aspects of report quality (0.471–0.766), and Likert scores for lesion information differed significantly between correctly and incorrectly classified lesions (all p ≤ 0.04). Repeatability varied widely from 12.7% (Med-LMop) to 39.0% (GPT-4sp).

Conclusions

Med-LM outperforms GPT-4 in classifying liver lesions from unstructured CT reports with both models better at detecting malignancy than full LI-RADS classification. However, high misclassification rates and inconsistent repeatability hinder their clinical use.