Power comparison of ANOVA and Kruskal‒Wallis tests when error assumptions are violated

Abstract

The effects of the violations of normality and homogeneity of variances assumptions on the power of the one-way ANOVA F-test is studied in this paper. Simulation experiments were conducted to compare the power of the parametric F-test with the non-parametric Kruskal‒Wallis (KW) test in normal/non-normal, equal/unequal variances scenarios and equal/unequal sample group means. Each of these 184 simulation experiments was replicated N = 1000 times and power obtained for both F and KW tests. The Shapiro‒Wilk's test for normality and Bartlett's/Levene's tests for homogeneity of variances was conducted in each experiment. Results show that the power of the KW tests outperformed those of the F-tests in the 92 (85/92) non-normal cases. Although the power of the F-tests is higher than those of the KW tests in 85 out of the 92 experiments under normality assumptions, these differences, in all cases in this study are not significant (p > 0.05) using both t and sign tests. Based on these results, this study favours the KW test as a more robust test and safer to use rather than the F-test especially when the distributional assumptions of data sets are in doubt.