Mapping the reliability multiverse of contextual cuing.

Abstract

Cronbach (1957) famously noted the divergence between the experimental and psychometric traditions in psychology and called for a unification, but many domains of cognitive experimental psychology continue to pay minimal heed to basic psychometric principles. The present article considers the lack of attention devoted to the reliability of measures extracted in a popular visual search task for studying putatively unconscious mental processes, contextual cuing, and the inferential fallacies that this neglect can cause. Two experiments (total N = 200) demonstrated that the reliability of contextual cuing and awareness measures can be increased by three manipulations designed to increase between-participant variability in search performance. At the same time, the data were subjected to a multiverse analysis, which found that specific data preprocessing pipelines result in more reliable estimates. Nevertheless, the reliability estimates remained too low for drawing firm conclusions from standard statistical techniques. Interpreting results from analyses based on individual differences, such as the typical low correlations between implicit and explicit measures, will be challenging so long as the underlying measures have poor reliability. (PsycInfo Database Record (c) 2024 APA, all rights reserved).