Projective Independence Tests in High Dimensions: the Curses and the Cures

Abstract

Summary Testing independence between high dimensional random vectors is fundamentally different from testing independence between univariate random variables. Take the projection correlation as an example. It suffers from at least three issues. First, it has a high computational complexity of O{n3 (p + q)}, where n, p and q are the respective sample size and dimensions of the random vectors. This limits its usefulness substantially when n is extremely large. Second, the asymptotic null distribution of the projection correlation test is rarely tractable. Therefore, random permutations are often suggested to approximate the asymptotic null distribution. This further increases the complexity of implementing independence tests. Last, the power performance of the projection correlation test deteriorates in high dimensions. To address these issues, we improve the projection correlation through a modified weight function, which reduces the complexity to O{n2 (p + q)}. We estimate the improved projection correlation with U-statistic theory. More importantly, its asymptotic null distribution is standard normal, thanks to the high dimensions of random vectors. This expedites the implementation of independence tests substantially. To enhance power performance in high dimensions, we introduce a cross-validation procedure which incorporates feature screening with the projection correlation test. The implementation efficacy and power enhancement are confirmed through extensive numerical studies.