Large language models for full-text methods assessment: a case study on mediation analysis.

Abstract

Objective: Systematic reviews remain labor-intensive, particularly when extracting methodological details from full texts. Using mediation analysis as a case study, we evaluated whether large language models (LLMs) can match human-expert-level full-text methodological review on key causal assumptions (eg, no unmeasured confounding, temporal ordering) and best practices (eg, sensitivity analyses, interaction assessments, covariate adjustment) for psychiatry and psychology studies.

Materials and methods: We evaluated 6 LLMs from 3 major families (ChatGPT-4o-mini/4o/o3/5, Claude Sonnet 4, Gemini 2.5 Flash) on 180 full-text mediation analysis articles from 2013 to 2018 previously reviewed by expert methodologists. LLMs assessed 14 binary methodological criteria ranging from straightforward checks (eg, whether the exposure was randomized) to nuanced assessments (eg, whether the temporal ordering between mediator and outcome was established). Performance was benchmarked against expert consensus labels and individual reviewers using accuracy, precision, recall, F1, AUC, and PR-AUC.

Results: LLM performance strongly correlated with human reviewers across methodological criteria (accuracy correlation 0.71; F1 correlation 0.95), indicating tasks difficult for humans were likewise challenging for models. Advanced LLMs achieved near-human accuracy on explicit methodological features but lagged behind top reviewers by up to 15% on inference-intensive tasks. Longer documents reduced model accuracy. Common model errors include overinterpreting on linguistic cues and colloquial use of technical terms.

Discussion and conclusion: Our findings support a criterion-specific human-AI collaboration strategy for full-text methodological assessment and provide a reproducible framework for future testing in other evidence-synthesis settings.