Including Pervasive Web Content in Evidence-Based Software Engineering: A Case Study

Context: Both scientific publications and grey literature have widely been employed as sources of empirical evidence in evidence-based software engineering (EBSE). However, there is still a fierce debate about whether or not the pervasive Web content can act as an alternative means to gather evidence for EBSE. Aim: To help ourselves enter this debate, this work aims to obtain some pre-evidence of reviewing Web documents for verifying the value and reliability of online materials. Method: Given the unique characteristics of Web content, we adapted the traditional Systematic Literature Review (SLR) methodology in EBSE, and conducted a review case study in the deep learning domain. Results: Our study selected four different search sources and captured 5082 "deep learning"-relevant Web documents. After a set of thematic synthesis steps ranging from keyword identification to brainstorming, the collected raw data were eventually evolved into a mind map of six semantic topics. Conclusions: We confirm that Web content can provide valuable information as supplementary evidence in EBSE. However, reviewing Web content introduces more search source bias rather than academic publications' location bias that is due to factors like ease of access or indexing levels in digital libraries.