Variability of Bayes Factor estimates in Bayesian Analysis of Variance

Bayes Factor estimation for Bayesian Analysis of Variance (ANOVA) typically relies on iterative algorithms that, by design, yield slightly different results on every run of the analysis. The variability of these estimates is surprisingly large, however: The present simulations indicate that repeating one and the same Bayesian ANOVA on a constant dataset often results in Bayes Factors that differ by a factor of 2 or more within only a few runs when using common analysis procedures. Results may at times even suggest evidence for the null hypothesis of no effect on one run while supporting the alternative hypothesis on another run. These observations call for a cautious approach to the results of Bayesian ANOVAs at present, and I outline three possibilities to circumvent or minimize this limitation.