The Impact on Systematic Reviews of Risk of Bias Assessment Changes From Conference Abstracts to Full Text

Abstract

Conference abstracts are commonly included in systematic reviews of evidence. Due to limitations in word count, conference abstracts often lack data or information. This causes issues for the assessment of risk of bias (RoB). We therefore aimed to compare the RoB rating, using the Cochrane RoB tool, for abstracts and full texts. This was accomplished using previously published Cochrane reviews and comparing RoB ratings for included studies originally included as an abstract and later as full text. To accomplish this, we searched the Cochrane Database of Systematic Reviews for reviews with updates across numerous disciplines (depression, anxiety, surgical, Parkinson's disease, Alzheimer's disease, multiple sclerosis, motor neuron disease, cancer, cardiovascular disease, and musculoskeletal disease). We identified 29 reviews, with 52 randomized controlled trials included, which had an abstract and subsequent full text available. If abstracts and full texts were not assessed using the Cochrane RoB tool, we obtained the texts and performed the assessment (n = 32). To assess the likelihood of changing the domain assessment rating (low, unclear, or high) from conference abstract to full text, we performed a Bayesian categorical multinomial model for each domain (i.e., signaling question) of the Cochrane tool. At the abstract assessment stage, the most common decision was unclear. Using unclear as the reference level in the model led to increased odds of being rated high at full text, compared to abstract assessment, for domains 2 (allocation concealment: odds ratio [OR] = 3.09, 95% credible intervals (CrI) 1.01 to 9.84) and 3 (blinding: OR = 5.09, 95% CrI 1.67 to 16.20). Domain 2 also had odds of being rated low (OR: 2.93, 95% CrI: 1.13 to 7.87). This suggests an impact of changing conference abstract to full text assessments on RoB. The numerous unclear ratings observed at the abstract assessment were usually due to a lack of reporting. While the findings of this study should be interpreted within the context of small numbers, the evidence still suggests that, in some instances, such as allocation concealment and blinding, it is likely that the decision could change based on full-text assessment. This also has implications for the certainty of the evidence, which is impacted by the RoB assessments, with having abstracts only or full texts available potentially changing the overall certainty. Current RoB tools may not be suitable for assessing conference abstracts.