Reporting completeness of clinical trials evaluating large language model interventions in medicine: systematic review.

Abstract

Objective: To characterise the reporting completeness of clinical trials involving large language model (LLM) interventions using established reporting guidelines.

Design: Systematic review and meta-research analysis.

Data sources: Bibliographic databases (Embase, Medline, and CENTRAL) were searched for trial reports and protocols, and trial registries (ClinicalTrials.gov and World Health Organization International Clinical Trials Registry Platform) were searched for trial registrations published from 1 January 2017 to 6 December 2025.

Eligibility criteria: Trial reports, trial protocols, and trial registrations describing clinical trials of LLM interventions in medicine. LLM interventions used for a healthcare function in a clinical trial setting included patient education, clinical documentation, communication support, triage, care navigation, management recommendations, or clinical decision support. Eligible studies prospectively assigned human participants to the intervention and evaluated a health related biomedical or behavioural outcome.

Main outcome measures: Overall study level and item level reporting completeness assessed using reporting guidelines matched to study design and purpose, including TRIPOD-LLM (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis-LLM), CONSORT (Consolidated Standards of Reporting Trials) 2025 and CONSORT-AI, SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) 2025 and SPIRIT-AI, CHART (Chatbot Assessment Reporting Tool), and the WHO Trial Registration Dataset (version 1.3.1). Two reviewers independently screened and extracted trial and intervention characteristics, and three calibrated raters assessed item level reporting completeness.

Results: The systematic search yielded 461 records, including 68 trial reports, 15 protocols, and 378 registrations. Trial reports were predominantly small sample, single site randomised studies published between 2024 and 2025, spanning diverse clinical specialties and primarily targeting patient facing education and management recommendations. Mean reporting completeness was modest across eligible trial reports (TRIPOD-LLM 40%, CONSORT 2025 44%, CONSORT-AI 59%, CHART 52%), trial protocols (SPIRIT 2025 51%, SPIRIT-AI 57%), and trial registrations (WHO Trial Registration Dataset 61%). Clinical trial reports of LLM interventions prioritise narrative framing and commentary over the methodological detail needed for replication, risk-of-bias assessment, and governance of model performance over time. Critical methodological elements for reproducibility, including data and intervention provenance, input preprocessing, model versioning and access modality, inference parameters, output postprocessing, as well as protocol, data, and code access, were inconsistently reported. Harms planning, monitoring, and reporting were often incompletely reported, limiting the critical appraisal of the safety of LLM interventions in healthcare.

Conclusion: Clinical trials of LLM interventions show modest adherence to established reporting guidelines, with persistent underreporting of LLM specific methodological details, reproducibility artefacts (protocols, data, code), and harms reporting required to support reproducibility and confident appraisal of safety risks before their clinical translation.