Comparing the Efficacy of Rapid Review With a Systematic Review in the Software Engineering Field

Abstract

Context

Rapid Reviews are secondary studies aiming to deliver evidence to experts in a more timely manner and with lower costs than traditional literature reviews. Previous studies have shown that experts and researchers are positive toward Rapid Reviews. However, little is known about how Rapid Reviews differ from traditional Systematic Reviews.

Objective

The goal of this paper is to compare a Rapid Review with a Systematic Review in terms of their methods (e.g., search strategy, study selection, quality assessment, and data extraction) and findings to understand how optimizing the traditional Systematic Review method impacts what we obtain with Rapid Review.

Method

To achieve this goal, we conducted a Systematic Review with the same research questions answered by a pre-existing Rapid Review and compared those two studies. Also, we surveyed experts from industry and academia to evaluate the relevance of the findings obtained from both the secondary studies.

Results

The Rapid Review lasted 6 days, while the Systematic Review took 1 year and 2 months. The main bottlenecks we identified in the Systematic Review are (i) executing the search strategy and (ii) selecting the procedure. Together, they took 10 months. The researchers had to analyze the information from 11,383 papers for the Systematic Review compared with 1973 for the Rapid Review. Still, most ($\sim$ 78%) of the papers included in the Systematic Review were returned by the Rapid Review search, and some papers that could be included were unduly excluded during the Rapid Review's selection procedure. Both secondary studies identified the same number of pieces of evidence (30), but the pieces of evidence are not the same.

Conclusion

The Rapid Review and Systematic Review results are inherently different and complementary. The time and cost to conduct a Systematic Review can be prohibitive in experts' contexts. Thus, at least in such situations, a Rapid Review may be an adequate choice. Moreover, a Rapid Review may be executed in the experts' context as a previous low-cost step before deciding to invest in a high-cost Systematic Review.