Artificial intelligence in systematic reviews and meta-analyses: Task-specific performance, residual error quantification, and human oversight

Abstract

Background

The volume of published research has expanded rapidly, intensifying the need for reliable, regularly updated evidence syntheses. Systematic reviews (SRs) and meta-analyses (MAs) remain the highest level of evidence, but their completion is time-consuming and resource-intensive, often extending beyond one year. Artificial intelligence (AI) is increasingly applied to automate stages of this workflow, yet validation of performance and methodological rigor remains limited. This narrative review addresses three questions: 1) at which stages of the SR/MA workflow can AI be used reliably; 2) how should AI performance be benchmarked for each task; and 3) how can the residual error remaining after human checking be quantified.

Methods

We outline the principles and key steps of a traditional meta-analysis and examine the contribution of AI at each stage. Literature was drawn from major biomedical databases; no systematic search was performed. AI performance is summarized by task, with benchmarking and residual-error quantification considered separately.

Results

Across the workflow, AI maturity is heterogeneous. In a 2025 systematic review of 19 comparative studies, generative AI missed 68.0%–96.0% of relevant studies in searching, made incorrect inclusion decisions in 0.0%–29.0% and incorrect exclusions in 1.0%–83.0%, incorrect data extractions in 4.0%–31.0%, and incorrect risk-of-bias assessments in 10.0%–56.0%. A 2025 scoping review identified 37 articles: 15/37 (41%) searching, 14/37 (38%) study selection, 11/37 (30%) extraction, 33/37 (89%) GPT-based, 21/37 (57%) validation studies; 20/37 (54%) drew a promising conclusion, none as a validated implementation. Title/abstract screening was most mature (sensitivity 99.2%, specificity 83.6%) versus full-text screening (97.6% and 47.4%). For structured extraction across 30 articles, one tool reached 92.0% precision, recall, and F1, but sensitivity fell to 77.0%–80.0% for review-specific variables, with confabulations in 4/90 (4%) of data points; across 107 trials, extraction accuracy was 96.2% (97.9% with refinement) while mean time fell from 86.9 to 14.7 min per trial. Risk-of-bias assessment was most fragile (kappa 0.51, 95% CI: 0.36−0.66). For residual error, 0 errors among 60 checked items corresponds to an upper one-sided 95% bound of 4.9%.

Discussion

AI is transforming SR/MA toward hybrid human-machine workflows rather than replacing them. Performance is task-specific, and metrics alone are insufficient: residual uncertainty should be quantified. Current evidence supports AI as an assistive technology under human oversight, not autonomous use.

Level of evidence

lV; narrative review.