What is the probability that higher versus lower quality of evidence represents true effects estimates?

Rationale, aims, and objectives: The previous studies demonstrated that the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system, a leading method for evaluating the certainty (quality) of scientific evidence (CoE), cannot reliably differentiate between various levels of CoE when the objective is to accurately assess the magnitude of the treatment effect. An estimated effect size is a function of multiple factors, including the true underlying treatment effect, biases, and other nonlinear factors that affect the estimate in different directions. We postulate that non-weighted, simple linear tallying can provide more accurate estimates of the probability of a true estimate of treatment effects as a function of CoE.

Methods: We reasoned that stable treatment effect estimates over time indicate truthfulness. We compared odds ratios (ORs) from meta-analyses (MAs) before and after updates, hypothesising that a ratio of odds ratios (ROR) equal to 1 will be more commonly observed in higher versus lower CoE. We used a subset of a previously analysed data set consisting of 82 Cochrane pairs of MAs in which CoE has not changed with the updated MA. If the linear model is valid, we would expect a decrease in the number of ROR = 1 cases as we move from high to moderate, low, and very low CoE.

Results: We found a linear relationship between the probability of a potentially 'true' estimate of treatment effects as a function of CoE (assuming a 10% ROR error margin) (R² = 1; p = 0.001). The probability of potentially 'true' estimates decreases by 21% (95% CI: 18%-24%) for each drop in the rating of CoE. A linear relationship with a 5% ROR error margin was less clear, likely due to a smaller sample size. Still, higher CoE showed a significantly greater probability of 'true' effects (53%) compared to non-high (i.e., moderate, low, or very low) CoE (25%); p = 0.032.

Conclusion: This study confirmed linear relationship between CoE and the probability of potentially 'true' estimates. We found that the probability of potentially "true" estimates decreases by about 20% for each drop in CoE (from about 80% for high to 55% for moderate to 35% to low and 15% to very low CoE).