The Pervasive Problem of Post Hoc Data Selection in Studies on Unconscious Processing.

Abstract

Studies on unconscious mental processes typically require that participants are unaware of some information (e.g., a visual stimulus). An important methodological question in this field of research is how to deal with data from participants who become aware of the critical stimulus according to some measure of awareness. While it has previously been argued that the post hoc selection of participants dependent on an awareness measure may often result in regression-to-the-mean artifacts (Shanks, 2017), a recent article (Sklar et al., 2021) challenged this conclusion claiming that the consideration of this statistical artifact might lead to unjustified rejections of true unconscious influences. In this reply, we explain this pervasive statistical problem with a basic and concrete example, show that Sklar et al. fundamentally mischaracterize it, and then refute the argument that the influence of the artifact has previously been overestimated. We conclude that, without safeguards, the method of post hoc data selection should never be employed in studies on unconscious processing.