Response to Letter to the Editor by Philip Good on To Permute or Not to Permute

Abstract

In current practice, such as GWAS (genome-wide association studies), permutation is often applied to multiple testing for association between large number of features [e.g. single nucleotide polymorphisms (SNPs)] and phenotypes (Hahn et al., 2008). Inferring that there is a difference between the phenotypic groups X and Y in some of the features is not very useful. One has to know for which features there is a difference. Exchangeability, a necessary condition for the validity of permutation tests, might be applicable if subjects are assigned randomly to treatments and the treatment is totally innocuous. However, instead of randomized, controlled clinical trials, bioinformatics discovery studies are mostly retrospective. Huang et al. (2006) gives examples of how permutation testing may fail to control Type I error when exchangeability does not hold. Equally important, Theorem 2.2 of this paper gives a succinct condition on when permutation testing is valid, even when exchangeability fails. This condition is as follows. In testing the null hypothesis that there is no difference in an entire set of features between groups X and Y , when the sample sizes are equal, even if the data distributions FX and FY have unequal even order cumulants, so long as they have equal odd higher order (third order and higher) cumulants, permutation testing controls Type I error rate. This precise condition is the basis for the subsequent papers Xu and Hsu (2007) and Calian et al. (2008) to uncover the Marginal-Determines-the Joint (MDJ) distribution condition needed for permutation multiple testing to control multiple testing error rates. Regardless of sample sizes, permutation multiple tests may not control false discoveries of which features are predictive of phenotype, unless it is assumed that the joint distributions of nonpredictive features are identical between the X and Y groups. Checking this assumption on the joint distribution using the data