Welch’s t test is more sensitive to real world violations of distributional assumptions than student’s t test but logistic regression is more robust than either

Abstract

It has previously been pointed out that Student’s t test, which assumes that samples are drawn from populations with equal standard deviations, can have an inflated Type I error rate if this assumption is violated. Hence it has been recommended that Welch’s t test should be preferred. In the context of carrying out gene-wise weighted burden tests for detecting association of rare variants with psoriasis we observe that Welch’s test performs unsatisfactorily. We show that if the assumption of normality is violated and observations follow a Poisson distribution, then with unequal sample sizes Welch’s t test has an inflated Type I error rate, is systematically biased and is prone to produce extremely low p values. We argue that such data can arise in a variety of real world situations and believe that researchers should be aware of this issue. Student’s t test performs much better in this scenario but a likelihood ratio test based on logistic regression models performs better still and we suggest that this might generally be a preferable method to test for a difference in distributions between two samples.

This research has been conducted using the UK Biobank Resource.