Reproducibility and validity failures in systematic review deduplication tools: evaluation on a paediatric neuroradiology corpus.

Abstract

Background: Deduplication is usually treated as a routine preprocessing step in systematic reviews, yet errors introduced at this stage are not visible in Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) flow diagrams and can propagate through evidence synthesis.

Objective: To evaluate the validity and reproducibility of commonly used deduplication tools in a paediatric neuroradiology systematic review.

Materials and methods: PubMed and Embase were searched for peer-reviewed literature on paediatric subacute sclerosing panencephalitis neuroimaging, yielding 603 records. Two reviewers independently deduplicated the combined corpus using Zotero (version 8.0.4) and Rayyan (updated 21 January 2026). A manual reference standard was established by dual-reviewer adjudication of potential duplicate pairs using title, publication year, author list, and Digital Object Identifier (DOI). Tool outputs were compared against this reference standard for false-positive and false-negative decisions and for reproducibility across runs.

Results: Manual adjudication identified 120 true duplicate records, leaving 483 unique records. Zotero produced identical outputs across reviewers, indicating deterministic behaviour, but generated 16 false-positive duplicate decisions and 5 false negatives, predominantly related to DOI collision. Rayyan identified more true duplicates but produced non-identical outputs across independent runs, with 16-17 false positives and 1 false negative per run.

Conclusion: Commonly used deduplication tools can introduce either deterministic bias or non-deterministic record loss before screening begins. In paediatric radiology evidence synthesis, the deduplication software version and verification strategy should be explicitly reported. Auditable secondary checks should be considered before records are removed.