Assessing the coverage of PubMed, Embase, OpenAlex, and Semantic Scholar for automated single-database searches in living guideline evidence surveillance: a case study of the international polycystic ovary syndrome guidelines 2023

Abstract

Objectives

Living guideline maintenance is underpinned by manual approaches toward evidence retrieval, limiting long-term sustainability. Our study aimed to evaluate the feasibility of using only PubMed, Embase, OpenAlex, or Semantic Scholar in automatically retrieving articles that were included in a high-quality international guideline - the 2023 international polycystic ovary syndrome (PCOS) guidelines.

Methods

The digital object identifiers (DOIs) and PubMed ID (PMIDs) of articles included after full-text screening in the 2023 international PCOS guidelines were extracted. These IDs were used to automatically retrieve article metadata from all tested databases. A title-only search was then conducted on articles that were not initially retrievable. The extent of coverage, and overlap of coverage, was determined for each database. An exploratory analysis of the risk of bias (RoB) of articles that were unretrievable was then conducted for each database.

Results

OpenAlex had the best coverage (98.6%), followed by Semantic Scholar (98.3%), Embase (96.8%), and PubMed (93.0%). However, 90.5% of all articles were retrievable from all four databases. All articles that were not retrievable from OpenAlex and Semantic Scholar were either assessed as medium or high RoB. In contrast, both Embase and PubMed missed articles that were of high quality (low RoB).

Conclusion

OpenAlex should be considered a single source for automated evidence retrieval in living guideline development, due to high coverage, and low risk of missing high-quality articles. These insights are being leveraged as part of transitioning the 2023 international PCOS guidelines toward a living format.