Lost in a large EEG multiverse? Comparing sampling approaches for representative pipeline selection

Background

The multiplicity of defensible pipelines for processing and analysing data has been implicated as a core contributor to low replicability, creating uncertainty about the robustness of results to defensible variations. This is exacerbated where many defensible pipelines exist, such as in processing electroencephalography (EEG) signals. In multiverse analyses, equally defensible pipelines are computed and the robustness across pipelines is reported. Computing all pipelines is often infeasible, and researchers rely on sampling approaches, assuming representativeness of the full multiverse. However, different sampling methods may yield different robustness estimates, introducing what we term multiverse sampling uncertainty.

New method

We developed an open-source tool to compare pipeline samples on their representativeness of the full multiverse. We computed a 528-pipeline use case multiverse on EEG recordings during an emotion classification task to predict extraversion scores from the Late Positive Potential. We applied three sampling methods (random, stratified, active learning) to sample 26 pipelines (5 %) and evaluated the representativeness of model fit distributions.

Results

Our results highlight variability in the representativeness of model fit distributions across samples, with active learning and stratified sampling most closely representing the full multiverse. Replicability of results is reported using cross-validation, and reproducibility is explored across pipeline sample sizes.

Comparison with existing methods

Large multiverse analyses in neuroimaging typically rely on sampling, but sampling approaches are not often systematically compared for their representation of the full multiverse.

Conclusions

The need for representative pipeline sampling to mitigate bias in large multiverse analyses is discussed.