MVPA significance testing when just above chance, and related properties of permutation tests

Parametric statistical tests (e.g., t-tests) can sometimes return highly significant results in cases that would be considered uninformative, such as when the individuals’ accuracies are just above chance. This paper demonstrates that permutation tests can produce the expected non-significant results in these datasets. The properties of null distributions underlying this difference in significance are illustrated: their relative insensitivity to dataset information content, but sensitivity to dataset characteristics such as number of participants, examples, and runs.