Multiple hypothesis testing for pre-screening of factor selection for classification of high-dimensional data

Modern instrumentation, such as mass spectrometry, enables the measurement of concentrations of hundreds or even thousands of compounds in individual samples. These measurements are often used in process data analytics to build classification models for determining whether a process is operating satisfactorily, if a product meets specifications, or to diagnose specific health conditions in patients. A common challenge associated with these applications is that the number of measured compounds far exceeds the number of available samples, increasing the risk of overfitting. Typically, it is advisable to have 10–20 samples per input factor of the classification model, thereby requiring the selection of only a handful of concentrations from potentially thousands. However, identifying the best combination of compounds from such a large pool by an exhaustive search is computationally infeasible.

A common approach to address this issue is pre-screening the compounds for statistically significant differences between groups, then limiting model inputs to only those identified as significant. The simplest form of pre-screening involves a student’s t-test, however, with a commonly-used $p$-value threshold of 0.05, one expects 5% of the compounds to be false positives, even when no true differences exist. Multiple hypothesis testing techniques, such as the Bonferroni correction and the Benjamini–Hochberg procedure, can reduce the number of compounds considered by accounting for these false positives. However, these methods often make assumptions about the data that are not valid in practice, leading to overly conservative results and potentially missing important compounds.

In this work, we present a screening procedure that computes the false discovery rate of p-values using a Leave-n-Out approach. By omitting $n$ samples at a time and repeatedly calculating the p-values, we assess the robustness of statistical significance against small changes in the dataset. We compare this technique to the Bonferroni correction and Benjamini–Hochberg procedure using both synthetic examples and two experimental datasets from the life sciences. Our results demonstrate that while the proposed approach is more conservative than a simple t-test, it identifies compounds that lead to better-performing models compared to those selected using existing multiple hypothesis testing methods.