Validity assessment of a computational system in the identification of duplicate studies

Abstract

Abstract Objective To evaluate the performance of the Systematic Review Support web-based system for the identification of duplicate records compared with similar software tools. Methods A methodological study was conducted assessing the automated process of de-duplication performed by the Systematic Review Support web-based system (version 1.0) versus the EndNote X9® and Rayyan® systems, adopting hand-checking as the benchmark reference for comparisons. A set of studies on three topics related to cystic fibrosis retrieved from the Pubmed, Embase and Web of Science electronic databases was used for testing purposes. The sensitivity, specificity, accuracy and area under the ROC curve of the software systems were compared to the benchmark values for performance evaluation. Results The database searches retrieved 1332 studies, of which 273 (20.5%) were true duplicates. The Systematic Review Support tool identified a larger proportion of true duplicates than the other systems tested. The sensitivity, specificity and accuracy of the Systematic Review Support tool exceeded 98%. Conclusion and implications for practice The Systematic Review Support system provided a high level of sensitivity, specificity and accuracy in identifying duplicate studies, optimizing time and effort by reviewers in the health field.