Experimenter evidence unmasking as a confound in optional stopping.

Abstract

Optional stopping refers to the practice of repeatedly performing a statistical analysis on a dataset as new data are collected until a pre-specified decision criterion is reached. This procedure is often adopted because of its effectiveness in optimizing data collection. Discussions of optional stopping to date have primarily centred around statistical issues, with relatively little consideration of any methodological implications of this procedure. Building on recent work drawing attention to methodological biases arising from the use of optional stopping, we highlight experimenter awareness of the current evidence state during data collection (experimenter evidence unmasking) as a salient methodological confound of optional stopping. We argue that experimenter evidence unmasking has the potential to influence an experimenter to implicitly or explicitly modify their behaviour in ways that can reduce the internal validity of an experiment. We conclude by offering recommendations for circumventing this confound and for the transparent reporting of experimenter evidence masking procedures.