Evaluating Large Language Models for imaging modality selection: Potential to reduce unnecessary contrast agent use and radiation exposure

Abstract

Introduction

Large Language Models (LLMs) represent a transformative leap in artificial intelligence with the potential to revolutionize radiologic decision-making. This study uniquely evaluates the performance of various LLMs from different vendors in selecting appropriate imaging modalities and comparing their responses with those of clinicians across different specialties and radiologists with different experience levels.

Methods

In a cross-sectional experimental design, 120 clinical scenarios derived from ACR AC and 120 “Multifaceted practice-oriented clinical scenarios” (including breast, cardiac, gastrointestinal, musculoskeletal, neuro, thoracic, genitourinary, vascular sections) were assessed using three different prompts. The performance of four LLMs from different vendors were evaluated and compared with four clinicians (emergency physician, cardiologist, internist and general surgeon) and four radiologists with different experience level. Also, the performances of LLMs contrast agent use and X-ray-containing imaging modality selection were evaluated. The responses were categorized according to ACR AC. Short and long-term reproducibility were assessed in the same clinical scenarios.

Results

All LLMs yielded identical modality recommendations across the three distinct prompts (κ = 1). In ACR clinical scenarios, DeepSeek-R1 identified the appropriate imaging modality in 98.3 % of cases, achieving superior accuracy without inter-model differences (p > 0.006). In realistic scenarios, DeepSeek-R1 again led, matching board-certified junior radiologist performance and exceeding clinician and resident performance. The short-term reproducibility ranged from κ = 0.773 to 0.886, with long-term reproducibility spanning κ = 0.507 to 0.787.

Discussion

This study underscores that LLMs have remarkable potential for selecting appropriate imaging modalities for different clinical scenarios related to various sections and their valuable contributions as supportive tools in clinical practice in this field.