Editor's Message: Teetering Atop the Pyramid: What to Do About Systematic Reviews?

Abstract

consequential problems with SRs have been recognized and discussed, sometimes heatedly, since the 1990s. These problems include, but are not limited to: redundancy, unimportant or overly broad research questions, insufficiently rigorous (and sometimes seriously biased or even deceitful) conduct of the review process, the inclusion of primary studies that suffer from bias and/or were not conducted well themselves, journal publication bias toward studies with significant results, substantial differences between included studies, and the cobbling together of fragmented evidence in order to produce an apparently seamless whole conclusion.10 A number of highly respected researchers have come to doubt the usefulness of SRs, with several arguing forcefully that SRs do more harm than good and should no longer be published.11 Other researchers counter that the quality of SRs has improved markedly, and these efforts to synthesize evidence for clinicians should not be abandoned.12 Many steps have been taken to improve the quality of SRs.13 Prospective registration of SRs is now required through PROSPERO or other registration sites. Research questions are guided by the PICO framework (Population, Intervention, Comparison, Outcome). Standardized guidelines for the procedural conduct of the SR study must be followed (e.g., PRISMA). Multiple authors are required, at least two who independently rate the candidate articles for inclusion and extract the data, with an additional author to resolve any discrepancies. Standardized tools to assess the quality of included articles (e.g., GRADE) and their risk of bias (e.g., Cochrane Risk of Bias – 2) are available and used for responsible reporting of the limitations of those included studies. Researchers supportive of the continued publication of SRs do acknowledge the many inherent problems. However, they posit that the need for evidence synthesis to support evidence-based clinical practice is real and growing, and that currently, no better alternative to SRs exists. These same authors pull no punches when it comes to SR quality, however, stating outright that editors should not publish un-registered or low-quality SRs.12 Based on my review of numerous SR submissions to the Journal of Geriatric Physical Therapy, the largest problem with SRs in the rehabilitation field is the sub-par quality of previously published clinical trials, a great number of which would not be accepted for publication by today’s standards. Too many have a very high risk of bias because there was no randomization, no assessor blinding, or indeed, no control group at all. The use of invalid, unreliable or unresponsive outcome measures often compounds the problems of interventions that do not specifically A systematic review [SR] is a secondary research study that uses defined, reproducible and transparent methods to locate, screen, and critically assess all previously published primary research studies designed to answer an important clinical question. From the articles included in a SR, pertinent data is described, summarized and analyzed.1 While SRs may cover observational studies, or address measurement topics, here I refer primarily to SRs of clinical intervention trials. About 1/3 of SRs include a metaanalysis [SRMA] in which statistical techniques are used to synthesize data from several similar studies in a single, summary quantitative estimate of the effect size of the studied intervention compared to either no intervention or a comparison intervention.2 The result of the meta-analysis of multiple studies is generally considered to be stronger evidence than the result of any single study, although this is not always the case. Overall, the publication rate for SR/ SRMAs in healthcare journals has increased dramatically, outstripping the publication rate for original clinical trials.3 This trend is concerning for reasons described elsewhere,3 however, this trend is not seen rehabilitation journals, where (at least until recently) clinical trials continue to be published at a higher rate than SRs.4 Systematic reviews are considered the highest level of evidence, and for this reason they are trusted sources of information to guide clinical decisions.5,6 They are popular with researchers, clinicians, and journals alike. Researchers investigating related topics value having a thorough sense of what is known and what is not known (knowledge gap) about a topic prior to designing and conducting their own studies. Clinicians striving for evidence-based practice who struggle to stay abreast of hundreds of published studies appreciate that an expert has done the leg-work and produced a useful summary for clinical decision-making. Journals seeking to improve their impact factor like the fact that SRs are frequently cited. What’s not to like, right? As it turns out, plenty. While there is little doubt that the benefits of highquality SRs are many, there is widespread consensus that poor quality SRs are at best misleading and thus potentially result in erroneous clinical decisions.7 Further, with the caveat that the quality of more recent SRs is increasing, extensive reviews of published SRs have found that a shockingly large majority are of low quality.8,9 Real and Editor’s Message: Teetering Atop the Pyramid: What to Do About Systematic Reviews?