The Citation of Retracted Papers and Impact on the Integrity of the Scientific Biomedical Literature

Abstract

A scientific paper may be retracted due to serious errors, misconduct, or fraud, including plagiarism, duplication, or fabrication, invalid authorship, copyright infringement, or the lack of ethical approvals (Steen, Casadevall, and Fang 2013; Bornemann-Cimenti, Szilagyi, and Sandner-Kiesling 2016; Dal-Ré and Ayuso 2019; Xie, Wang, and Kong 2021; Candal-Pedreira, Pérez-Ríos, and Ruano-Ravina 2022). Historically, about 20% of retracted papers involve collaborative research (Zhang, Abraham, and Fu 2020), affecting a network of individuals. Recently, there has been an increase in retractions linked to paper mills, which are paid services providing data, analyses, or even an entire paper, and whose use is unethical in scientific publishing if those services are undeclared (Rivera and Teixeira da Silva 2021; Day 2022; Pérez-Neri, Pineda, and Sandoval 2022; Santos-d'Amorim et al. 2024; Teixeira da Silva et al. 2024). These papers are retractable due to the undeclared use of such services and the potential inclusion of fake or fabricated data.

The number of retractions has amplified considerably from 38 in 2000 to over 2300 in 2020 (Oransky et al. 2021), reaching a new record in 2023 with over 10,000 articles retracted (Van Noorden 2023). Figure 1 displays the steady increase of publications labelled as retracted in the PubMed, Scopus, and Web of Science (WoS) databases over the past 30 years. One reason why more literature is not retracted are legal threats to publishers by authors (Sox and Rennie 2006; Oransky 2022). Moreover, a negative stigma and the punitive nature of retractions (Teixeira da Silva and Al-Khatib 2021; Xu and Hu 2024), due to a name-and-shame style culture (Hu and Xu 2023), may stifle academics' desire to correct the literature (Vuong 2020) because retractions can be career-altering or career-terminating events (Mongeon and Larivière 2016; Hu and Xu 2020; Lin 2024). Opinion papers might also be retracted, but if personal or political biases are at play, then such retractions may form part of ‘cancel culture’ in academia (Teixeira da Silva 2021).

There is no clear or standard response to this question (Teixeira da Silva 2020), possibly explaining why retracted papers continue to be cited, even though they should theoretically not be (Hansson 2023). However, an answer may lie in the reason why a paper was retracted rather than why it was corrected (Fanelli, Ioannidis, and Goodman 2018). Thus, if a retraction is due to misconduct, then it should not be cited (Neale et al. 2007; Neale, Dailey, and Abrams 2010). The Committee on Publication Ethics (COPE) advises that ‘[a]rticles that relied on subsequently retracted articles in reaching their own conclusions, such as systematic reviews or meta-analyses, may themselves need to be corrected or retracted’ (COPE 2019). This suggests that the citation of a retracted paper must be individually and carefully considered, especially its use in systematic reviews that provide a concise summary of a topic and that rely on factually accurate and robust information, so as not to mislead their readers (Gray et al. 2018; Faggion Jr. 2019; Herrera-Peco et al. 2020; Shi et al. 2021; Fanelli, Wong, and Moher 2022).

Assuming that a retraction represents collective failure on the part of authors, peer reviewers, editors, and/or publishers (Teixeira da Silva 2016a), then one response might be to not cite any retracted literature to avoid any risk to one's reputation, and to not cite erroneous or fraudulent literature (Stern et al. 2014). The expectation is that citations to retracted literature, and attention to papers retracted due to misconduct, should decrease over time (Mistry, Grey, and Bolland 2019; Mott, Fairhurst, and Torgerson 2019).

Retraction notices (RNs) are intended to clarify the reasons and background behind retractions but often fail to provide comprehensive explanations (Vuong 2020; Teixeira da Silva and Vuong 2022; Xu and Hu 2022a), most likely because many stakeholders are involved (Xu and Hu 2022b, 2022c). Content variation in RNs tends to occur due to factors such as editorial freedom (Lundberg 1988), unclear ethical guidelines (Bhargava, Vaswani, and Vaswani 2019), potential litigation threats (Moylan and Kowalczuk 2016), the absence of academics in policy-making, or excessive reliance on policy groups (Teixeira da Silva and Vuong 2022). Moreover, the language used in such RNs may be ambiguous, deceptive, or lacking insufficient factual support (Lin and Chen 2022; Lin 2024). Consequently, readers and academics who are meant to be educated by such RNs, so that they can make a decision about whether to cite a retracted paper, are not being properly or fully informed about the risks.

A citation, regardless of its validity, serves as a ‘reward’ by recognising the cited paper. However, citations often fail to distinguish between ‘positive’ and ‘negative’ recognition (Bordignon 2020). Some retracted papers accrued more post-retraction citations than before their retraction (Bolboacă et al. 2019; Schneider et al. 2020), suggesting an imperfect mechanism of communication about retractions. Thus, insufficient correction of scientific literature is a driver of citation inflation that can distort impact indicators (Teixeira da Silva and Dobránszki 2018a, 2018b; Dobránszki and Teixeira da Silva 2019; Teixeira da Silva and Vuong 2021).

Citing a retracted paper can negatively impact a research field, particularly if widely cited, as it may invalidate claims or findings of subsequent papers (Chen et al. 2013; De Cassai et al. 2023). Post-retraction citations may be intentional or inadvertent, and readers might not always be aware of the reasons why the authors cited those papers (Teixeira da Silva and Bornemann-Cimenti 2017). In a survey, 89% of corresponding authors of papers that cited retracted articles on anaesthesiology and intensive care medicine were not aware of the retracted status of those papers (De Cassai et al. 2022). Of the greatest concern are academics that cite papers that were retracted due to misconduct (Bar-Ilan and Halevi 2017).

Retracted literature continues to be cited for several reasons. Authors may believe that their retracted work remains scientifically valid (Madlock-Brown and Eichmann 2015; Teixeira da Silva and Dobránszki 2016), or several findings might still remain valid despite retraction (Azoulay et al. 2015). The retracted status of a paper might not be indicated or visible on the journal websites (Hesselmann et al. 2017; Deculllier and Maisonneuve 2018; Boudry, Howard, and Mouriaux 2023) or scientific databases, including Scopus or WoS (Suelzer et al. 2021). Citing authors may use original (unretracted) copies from the publication's website (Elia, Wager, and Tramèr 2014; Frampton, Woods, and Scott 2021) or archived versions in personal libraries, pirate open access (OA) websites like Sci-Hub, or academic social networks sites, such as Academia.edu or ResearchGate (Davis 2012; Teixeira da Silva and Dobránszki 2017a; Binning, Jutfelt, and Sundin 2018). Reference managers such as Zotero and Endnote aim to reduce such citations by integrating comprehensive databases, such as the Retraction Watch database (Cheng et al. 2019; Dinh, Cheng, and Parulian 2019). By synchronising publications' metadata, it is possible to flag retracted references (Schneider et al. 2022). Although the integration of these tools can be a positive strategy to avoid citations to retracted documents, information in reference managers or bibliographic platforms might be inaccurate or outdated (Suelzer et al. 2021), not indicating a paper as retracted (Bakker and Riegelman 2018; Schmidt 2018), or the ‘retracted’ watermark on the pages of the PDF or HTML text of a retracted paper may be missing (Gray, Al-Ghareeb, and McKenna 2019).

Besides these structural reasons, there may also be behavioural factors. Authors might not verify the status of papers prior to submission or might consider the retracted paper's methodology or findings as valid and thus citable (Teixeira da Silva and Bornemann-Cimenti 2017). Authors may intentionally cite retracted papers, for example, in publications that focus on fraud, misconduct, or retractions. Another possibility is when a retracted publication is reproduced or republished after correcting it (Cagney et al. 2016).

Several of these reasons explain why some retracted papers continue to be highly cited (Teixeira da Silva and Dobránszki 2017a; Heibi and Peroni 2021). Avenell et al. (2024) suggested that editors and peer reviewers screen the Retraction Watch database of retractions and expressions of concern (EoCs) to assess whether cited literature has been retracted or has an associated EoC. However, EoCs can sometimes take weeks, months, or years to resolve, leaving authors undecided about citing that work (Teixeira da Silva 2023a).

Citations link papers from any source, and this is a foundation of knowledge accumulation in academic publishing (Azoulay et al. 2015). A retraction disrupts the trust and legitimacy of this knowledge foundation (Furman, Jensen, and Murray 2012). Erroneous or fraudulent papers cited disseminate correspondingly erroneous or fraudulent information, infecting a network of surrounding literature (Teixeira da Silva 2024). The academic community is more sensitised to this issue when such papers involve medical information (Budd et al. 1999) since the assumption is that the health of the public (including patients) might in some way be at risk of harm if the retracted data serve as the basis for further research (Steen 2011; Marcus, Abritis, and Oransky 2022; Teixeira da Silva 2022a) because, in clinical practice, evidence forms the basis of clinical governance (Dawes and Sampson 2003; Mikalef, Kourouthanassis, and Pateli 2017; Khademizadeh et al. 2024). Authors that actually read and cite such papers do so consciously, but if citations are planted into a paper, for example, by paper mills, then damage the scientific literature's integrity is two-fold: first, the illicit use of a paper mill; second, the ‘plantation’ of citations into a paper for a price. In the latter case, editors, copy editors, journals, and publishers must complete thorough quality control prior to publication to (1) ensure that retracted literature is not cited or, if it is, that its citation is validated and (2) identify citations that are irrelevant to the content of a paper. If a paper that cites an erroneous or fraudulent paper is not corrected (Teixeira da Silva 2015), then subsequent citations (primary, secondary, tertiary, etc.) cumulatively ‘reward’ it (Bar-Ilan and Halevi 2018).

In the worst-case scenario, a highly cited erroneous or fraudulent paper ‘infects’ a large body of literature, thereby biasing its information (Teixeira da Silva and Dobránszki 2017a). Papers retracted for plagiarism or publisher error do not negatively impact ‘surrounding’ literature as much as those retracted for fraud and misconduct, and the potential damage is greater when retracted papers are cited in ‘hot’ fields of research (Azoulay et al. 2015) or in reputable journals or publishers, such as Nature, Science, and Cell (Wang and Su 2024), that is, those that claim to adhere to community or industry standards (e.g., peer review), codes of ethical conduct (e.g., COPE or International Committee of Medical Journal Editors (ICMJE) guidelines), indexing (WoS, Scopus, PubMed/Medline, etc.) or metrics (Clarivate impact factor (IF), Elsevier/Scopus CiteScore).

As a theoretical example, a fraudulent paper published in a COPE-adhering, Scopus-indexed and IF-carrying journal, if retracted but cited by 100 similar quality journals, corrupts the literature 100-fold at the first level of citation due to the citation network (van der Vet and Nijveen 2016), despite adherence to ethics, indexing and quality control controls. Even the five-fold citation of 1000 papers may impact as many as 5000 papers (primary impact), and even more papers that cite the citing paper (secondary, tertiary, etc., impact), creating a ‘domino effect’ (Leta, Araujo, and Treiber 2022). However, dominoes imply a knock-down or erasure effect, akin to losing citations over time (Yang et al. 2024). Therefore, we prefer to think of the impact of amplified citation to retracted literature more like the growth of bacterial or fungal colonies that continue to propagate provided that they are offered sustained nutrients, which, in this imagery, is academic interest or reliance. Downstream citations can thus provide an unfair citation advantage (Schneider et al. 2020), and the extent to which that retracted literature distorts the scientific validity of downstream citing papers by biasing their scientific validity can be seen as an ‘epistemic’ cost (Fanelli, Wong, and Moher 2022).

With tens of thousands of papers retracted in biomedical science (Oransky 2022), the literature becomes a patchwork of erroneous and fraudulent information (Teixeira da Silva 2024), blurring the lines between valid and invalid, accurate and inaccurate, and truth and falsehood. Moreover, literature should not be permanently deleted—despite arguments to the contrary by some (Kühberger, Streit, and Scherndl 2022; Rzymski 2022)—as this may constitute poor librarianship and disrupt the permanence of scientific and biomedical knowledge, although the archival of erroneous literature ensures its permanence and its effective amplification (Davis 2012).

At some point, the literature might become irretrievably corrupted if the volume of retracted papers exceeds a threshold. At that point, even if there are measures to correct the literature (Teixeira da Silva 2022b), the sheer mass of negatively impacted literature may become so large that trust, an essential aspect of publishing, may become irretrievably lost. Consequently, as the mass of retracted papers on platforms such as PubMed accumulates, their reliability as sources of biomedical literature diminishes (Teixeira da Silva 2023b). The same applies to WoS (Fahimifar, Ghorbi, and Ausloos 2022), Scopus (Bolland, Grey, and Avenell 2022), Google Scholar, and other major private or public databases that serve as reliable sources of scientific information (Fanelli, Wong, and Moher 2022). Highly cited papers can extensively ‘corrupt’ the literature if they contain erroneous or misleading information (Heibi and Peroni 2021). Uncorrected citations to fraudulent literature contribute to misinformation, and deliberate actions amount to disinformation. What, then, is the threshold or point-of-no-return when the mass of such papers critically influences the integrity of the indexed information or the reliability of the index itself?

Ultimately, consciously or not, even that has been retracted can be promoted through citation, and it is the responsibility of editors, journals, and publishers to ensure that literature is corrected. Absent proper and responsible editorial action to address this issue, and to inform and educate the academic community about a paper's flaws that led to its retraction, via transparent RNs, journals and/or their information management may be relegated to the status of ‘junk’ because it involves an irresponsible approach towards the integrity of the knowledge base (Teixeira da Silva 2023c).

We dwell momentarily on the issue of how the scientific literature is corrected and the mechanisms currently in place to correct it. These mechanisms are constantly evolving (Teixeira da Silva 2016b; Teixeira da Silva and Dobránszki 2017b), in some ways adjusting to meet the demands and challenges that face academic publishing in the light of new forms of fraud, risks, and threats, seeking novel ways to decipher honest from dishonest error, mild from serious errors, or error from misconduct or fraud (Xie, Wang, and Kong 2021; Teixeira da Silva 2022b). These mechanisms are still imperfect, and they are still rigid and inflexible because they seem to be unable to accommodate a context of continually changing literature that results from new challenges, including through the actions of post-publication peer review (Yeo-Teh and Tang 2023).

Despite the existence of structural guidelines to address and correct erroneous or fraudulent literature, institutional investigations may be slack, lacking, tardy, or incomplete (Grey, Bolland, and Avenell 2019; Xu et al. 2023), while journals do not always follow best practices, leading some to claim that the system of correction is ‘broken’, due to inefficiencies, lack of timeliness and transparency, inconsistencies in policy implementation, inadequate editorial responses, and/or procrastination, even among COPE members (Wiedermann 2018; Bolland et al. 2021; Grey, Avenell, and Bolland 2022, 2024). To address this, a double DOI-based ‘publication history’ formally and permanently documents the paper's history, including errors and retraction-worthy information (Teixeira da Silva and Nazarovets 2022). This promotes transparency in the science and enrich readers' understanding of the historical background of that paper, both positive and negative. It is not helpful that one of the mechanisms, EoCs, are not always rapidly, efficiently, or decisively resolved (Teixeira da Silva and Nazarovets 2024), leaving academics who wish to cite papers with an associated EoC undecided as to whether those papers are scientifically reliable, and whether they should be cited (Teixeira da Silva and Yamada 2021; Grey, Avenell, and Bolland 2022). Whereas a RN alerts readers not to cite work during their scientific inquiry, an EoC serves as a retraction preamble, alerting readers to cite with caution or to critically evaluate information prior to use (Vaught, Jordan, and Bastian 2017). As advised by the ICMJE, editors and peer reviewers should screen references in submitted papers for retracted literature, while authors should declare, upon submission, the use of retracted literature or provide an explanation as to why such citation is necessary (ICMJE 2025).

This section highlights selected studies that conducted a bibliometric analysis of retracted papers in biomedical literature. These cases are presented in chronological order, except where studies are related, in which they have been clustered. This phenomenon, as a science in itself, already has a long history, with concerns having already been expressed over three decades ago (Lindberg 1987; Pfeifer and Snodgrass 1990; Kochan and Budd 1992). Bibliometric studies enhance an understanding of the problem and emphasise that despite the crucial process of correcting the scientific record through RNs, retracted articles in the biomedical field continue to be cited.

The field of biomedicine witnessed the first studies on post-retraction citation analysis (Schneider et al. 2020). Garfield and Welljams-Dorof (1990) analysed citations from 20 publications by Stephen E. Breuning between 1980 and 1988, using Science Citation Index and Social Sciences Citation Index data. They identified the prevalence of neutral citations—without judgement of their scientific validity—noting that of the 200 collective citations, 80 (40%) were self-citations. Twenty-four years after their retraction, those articles continued to be cited (Korpela 2010). Pfeifer and Snodgrass (1990) noted that although a retraction event decreased the number of citations, 75.3% of retracted articles continued to receive citations, even one year after the retraction.

Nath, Marcus, and Druss (2006) observed a higher probability of retractions due to unintentional errors than misconduct among retracted articles indexed in MEDLINE between 1982 and 2002. In a high-profile case, Scott Reuben's papers (n = 27) with allegedly fabricated data were cited in as many as 25 systematic reviews, and one in six might have reached different conclusions had they not included such studies (Marret et al. 2009). By 2014, 20 of Reuben's 25 retracted papers had accumulated 274 citations (Bornemann-Cimenti, Szilagyi, and Sandner-Kiesling 2016). Of the 420 citations accrued in a 10-year period (2009–2019) for 10 retracted Reuben papers, only 40% of the sources citing these retracted papers indicated their retracted status (Szilagyi et al. 2022). Using WoS citation data indexed between 1993 and 2001, Neale, Dailey, and Abrams (2010) found no differences between the volume of citations to articles affected by misconduct relative to articles in the control group. Using PubMed data, Foo (2011) identified repeat authors in misconduct practices, noting that the availability of information in online databases helped to detect fraudulent publications. Lu et al. (2013) noted an average annual loss (6.9%) in citations by a body of 1465 WoS-indexed retracted papers published post-2000. From PubMed and Medline data between 2004 and 2013, Singh et al. (2014) also identified that retracted papers continued to be cited, exemplified by an article published in Nature that was retracted only after 19 years, accumulating 255 citations. That study also revealed that the time to retraction had decreased, consistent with the findings of Furman, Jensen, and Murray (2012). Azoulay et al. (2015), studying 1104 retracted biomedical papers indexed on PubMed, published in 2007 or earlier and retracted no later than 2009, noted a 6% decline in citations relative to a body of 110,000 non-retracted papers. Of 81 retracted English dental publications that were cited by 685 papers, only 5.4% of them indicated the retracted state of those papers (Theis-Mahon and Bakker 2020).

Avenell et al. (2019) queried (Avenell et al. 2009; Bolland et al. 2016) the results of 33 reports of randomised controlled trials (RCTs) from a group in Japan that led to, at the time of publication of that paper, 27 retractions. Focusing on this subset and scrutinising only those papers published between 1997 and 2011 linked to RCTs with hip fractures as an outcome and published in journals with an IF > 4, they first noted that the 12 analysed papers were all retracted between June 2016 and April 2019, but collectively accrued 1158 citations by August 2016. Avenell et al. (2019) discovered 40 papers that cited these 12 papers, but 85% did not show any concern about the RCT reports. Avenell et al. (2019) claimed that the conclusions drawn in 41% (13/32) of them would likely have changed. Hagberg (2020) analysed retraction cases involving Eric Poehlman, a biomedical researcher who falsified data, noting that while the citation rate of his retracted papers decreased, it was not significantly different from his non-retracted papers. Schneider et al. (2020) found that a retracted article on a nutritional medical intervention continued to be cited even 11 years after the retraction. Similarly, Audisio et al. (2022) found that retractions in the cardiovascular field mainly occurred in journals with an IF < 5. Kohl and Faggion Jr. (2024) analysed 162 articles indexed in PubMed, WoS, and Scopus, identifying plagiarism as the most common reason for retraction and highlighting their poor methodological quality. Pinho-Gomes, Hockham, and Woodward (2023) reported that 60.9% of retractions in biomedical research involved men as the first and last authors in articles indexed in Retraction Watch between 1970 and 2022.

Brown, Bakker, and Theis-Mahon (2022) noted that 283 of 1396 retracted papers were cited 1096 times in systematic reviews, with 65% of these citations occurring before retraction. Similarly, Dal-Ré and Ayuso (2021) found that 23% of 34,487 citations to 460 retracted genetics papers accrued post-retraction. In contrast, Candal-Pedreira et al. (2020), focusing on 304 retracted articles indexed in MEDLINE, noted a general post-retraction increase in citations relative to the pre-retraction period, with exceptions. Campos-Varela, Villaverde-Castañeda, and Ruano-Raviña (2020) found misconduct cases more frequent in lower IF journals. Similarly, Audisio et al. (2022), who studied Retraction Watch-indexed retractions in the cardiovascular field between 1978 and 2020, found that most retractions occurred in journals with an IF < 5. Kohl and Faggion Jr. (2024) investigated the methodological quality of 162 articles indexed in PubMed, WoS, and Scopus, highlighting plagiarism as the most prevalent reason, and calling out that the suboptimal methodological quality of those studies. Regarding gender differences, Pinho-Gomes, Hockham, and Woodward (2023) noted that 60.9% of retractions in biomedical research in articles indexed in Retraction Watch in 1970 and 2022 had men as the first and last authors. While retraction broadly signals an article's lack of scientific validity, the increase of post-retraction citations has prompted studies on the obsolescence of retracted papers, creating a paradox. The extent of the half-life phenomenon on the citation of retracted papers remains uncertain (Berenbaum 2021). Widely acknowledged citation standards might differ depending on the research field, subfields, journals, and document types (Glänzel and Schoepflin 1994; Larivière, Archambault, and Gingras 2008; Faber, Eriksen, and Hammer 2021), adding complexity to this issue.

A wide range of errors may plague the scientific literature (Casadevall, Steen, and Fang 2014, ranging from unintentional typographic errors at the lower level of an ‘integrity’ scale to intentional falsification and fabrication at the upper end (Bolland et al. 2021). Except for academic purposes (Hsiao and Schneider 2021) or bibliometric analyses, the citation of retracted literature is likely not a wise scholarly choice. This is because those papers were retracted due to some infraction of a stated rule or ethical policy. There are ample reasons why academics continue to cite retracted literature, one of them being the belief that information in such papers is valid and thus free to cite. Stringent policy on how such literature should be cited and how such cited literature should be corrected is needed (Schneider et al. 2022). Publishers also need to be proactively involved in the post-publication process associated with the correction of the literature (Toma, Padureanu, and Toma 2022). Finally, we do not advocate for the deletion of retracted papers from the bibliometric record, that is, ‘hard retractions’ (Rzymski 2022), even though this has been proposed as one way for science to self-correct (Kühberger, Streit, and Scherndl 2022).

The authors contributed equally to literature searches, writing, and editing.

The authors declare no conflicts of interest.