Phantom citations: An empirical study of non-existent and unverifiable references in scholarly literature

Abstract

The integrity of scholarly communication depends critically on the accuracy and verifiability of cited references. Citations enable readers to trace prior work, assess evidence, and situate new contributions within the existing literature. However, concerns have emerged regarding the presence of references in published papers that cannot be resolved to any identifiable source, including citations that appear structurally complete yet correspond to non-existent publications. While such problems have long been recognized in the form of citation errors and fabricated references, recent developments in automated text generation using large language models have renewed attention on the reliability of bibliographic data in academic publishing. This study addresses two research questions: Can we develop an automated, auditable tool to detect non-existent or unverifiable references in scholarly manuscripts? What is the prevalence and distribution of such unverifiable references in peer-reviewed papers at the intersection of artificial intelligence and education published since 2023? To answer these questions, we develop a reference verification pipeline that combines reference extraction with DOI/URL checks and cross-database matching, and apply it to a corpus of 3201 peer-reviewed papers (published between 2023 and 2025). We flag 69 papers (2.16%) that contain at least one reference that remains unverifiable under our protocol; such unverifiable references can mislead readers and propagate through citation chains. This suggests that greater care is needed when using, reviewing, and reusing bibliographies.