Bonferroni’s method, not Tukey’s, should be used to control the total number of false positives when making multiple pairwise comparisons in experiments with few replicates

Statistical tests can be used to help determine whether experimental manipulations produce effects. In tests of means, when more than two groups are compared the total number of Type 1 errors (false positive results) increases unless a correction is used. Tukey’s method is thought to offer good control of the number of false positives and high statistical power when all pairwise comparisons are made. However, the number of replicates in laboratory experiments is often quite low, and small sample sizes can undermine assumptions underlying statistical methods. I used simulations to investigate how well ANOVA followed by different post-hoc tests controls the total number of false positives when there are 3- 6 experimental groups and 2- 6 experimental replicates, conditions that span the range of typical values. Tukey’s method, one of the most common, allows too many. I investigated 11 other approaches to controlling false positives and found that none is as effective as the simple Bonferroni correction or offers much more power. I conclude that researchers should not make all pairwise comparisons using ANOVA followed by Tukey’s method but instead use Bonferroni’s method on a limited number of pre-selected comparisons.