Use and misuse of corrections for multiple testing

Abstract

Current psychological research addresses multifaceted questions demanding multiple analyses of data. Statistical analyses regarded as instances of multiple testing are often subjected to alpha adjustments to guard against inflation of Type-I errors. A review of papers published in the last two years in two major psychology journals shows inconsistent and discretionary use of alpha adjustments in a broad diversity of statistical analyses that are formally identical across papers. Authoritative sources also do not clarify the circumstances in which alpha adjustments should or should not be used. This paper describes the workings of Bonferroni and false-discovery-rate adjustments, showing that they only control the Type-I error rate for an (omnibus) hypothesis stating that all its individual (surrogate) nulls are true. For individual nulls, alpha adjustment only has the trivial consequences of the use of a lower alpha level, without reducing the occurrence of Type-I errors or Type-II errors below their expected rates. In practice, then, corrections for multiple testing only come down to testing individual hypotheses at a lower alpha level without preventing the rejection of true nulls and without favoring the rejection of false nulls. Thus, use of alpha adjustments is only justifiable for inferences about an omnibus null for which a one-shot statistical test does not exist and which must instead be tested piecewise via several surrogates that collectively speak about the omnibus null. Recommendations for the use and reporting of alpha adjustments are given for a variety of statistical analyses with which they are often implemented.