Adding relevance to rigor: Assessing the contributions of SLRs in Software Engineering through Citation Context Analysis

Abstract

Research in Software Engineering greatly benefits from Systematic Literature Reviews (SLRs), in view of the citations they receive. While there has been a focus on improving the quality of SLRs in terms of the process, it remains unclear if this emphasis on rigor has also led to an increase in relevance. This study introduces Citation Context Analysis for SLRs as a method to go beyond simple citation counting by examining the reasons behind citations. To achieve this, we propose the Resonance Scheme, which characterizes how referring papers use SLRs based on the outputs that SLRs can provide, either backward-oriented (such as synthesis or aggregating evidence) or forward-oriented (such as theory building or identifying research gaps). A proof-of-concept demonstrates that most referring papers appreciate SLRs for their synthesis efforts, while only a small number refer to forward-oriented outputs. This approach is expected to be useful for three stakeholders. First, SLR producers can use the scheme to capture the contributions of their SLRs. Second, SLR consumers, such as Ph.D. students looking for research gaps, can easily identify the contributions of interest. Third, SLR reviewers can use the scheme as a tool to assess the contributions that merit SLR publication.