GPT-4o in radiology: In-context learning based automatic generation of radiology impressions

Abstract

Translating radiological findings into clinical impressions is critical for effective medical communication but is often labor-intensive and prone to variability. This study investigates the potential of the GPT-4o large language model (LLM) to automate the generation of radiology impressions from reports, using in-context learning techniques to improve accuracy. Using the MIMIC-IV-CXR dataset, the study compares three generative AI approaches: zero-shot generation (ZS), in-context learning with random examples (ICLR), and in-context learning with semantic nearest neighbors (ICLSN). These methods were evaluated using text summarization metrics such as BERT Score, ROUGE, and METEOR. Statistical tests, including the Kruskal–Wallis and Mann–Whitney U tests, were employed to validate the results. The ICLSN approach significantly outperformed ZS and ICLR, achieving the highest precision (0.9002 ± 0.0471), recall (0.8914 ± 0.0501), and F1 scores (0.8952 ± 0.0432) according to BERT Score. ROUGE and METEOR metrics confirmed these findings, with ICLSN showing notable improvements in ROUGE-1, ROUGE-2, and ROUGE-L scores (0.4673 ± 0.2606, 0.3130 ± 0.2863, and 0.4198 ± 0.2674, respectively). METEOR scores also improved significantly with ICLSN (0.4448 ± 0.2804). The study demonstrates that GPT-4o, particularly when using semantic nearest neighbors for in-context learning, can effectively generate clinically relevant radiology impressions. The method enhances the accuracy and reliability of automated clinical text summarization, suggesting a valuable tool for improving the efficiency and consistency of radiological assessments. Future work should explore fine-tuning to further optimize these outcomes and extend applications to other clinical texts.