On the Limitations of Manipulation Checks: An Obstacle Toward Cumulative Science

Abstract

Manipulation checks do not allow ruling out or accepting alternative explanations of causal effects (Sigall & Mills, 1998). In order to gauge the influence of this argument on current research practices, we surveyed the views of researchers on manipulation checks. Results confirmed that a manipulation check still stands as a totem of experimental rigor. Except in rare circumstances, such as when pilot testing, manipulation checks do not provide information relevant to construct validity. While it seems cost free to include seemingly informative manipulation checks, we claim it is actually costly because it wrongly enhances subjective confidence in the validity of research findings. We conclude that manipulation checks may hinder efforts to adopt a cumulative culture and practice of hypothesis testing.