A cautionary note against selective applications of the Bayes factor.

Abstract

Bayes factor analysis becomes increasingly popular, among other reasons, because it allows to provide evidence for the null hypothesis, which is not easily possible with the traditional frequentist approach. A conceivable strategy that apparently takes favorable aspects of both approaches on board is to use traditional frequentist analyses first and to support theoretically interesting nil effects by Bayesian analyses thereafter. Here, we asked whether such a selective application of Bayesian analyses to only nonsignificant effects of foregoing frequentist analyses creates bias. In two simulation studies, we observed that such selective application of Bayesian analyses, in fact, severely overestimates evidence in favor of the null hypotheses, when a true population effect exists. While this bias can be attenuated by using more informative priors in the Bayesian analyses, we recommend to not apply such selective combination of analytical approaches, but instead to use either frequentist or Bayesian analyses consistently. (PsycInfo Database Record (c) 2024 APA, all rights reserved).