A simple method for improving generalizability in behavioral science: Scope Testing with AI-Generated Stimuli (STAGS)

Abstract

Behavioral research typically tests hypotheses in a limited set of researcher-selected contexts. This approach can reveal whether an effect can occur (possibility) but does not indicate whether it holds in other contexts (generalizability). We present Scope Testing with AI-Generated Stimuli (STAGS), a simple approach that uses Generative AI (GenAI) to test predictions across a range, or scope, of stimuli. By assessing whether a prediction holds across this range, STAGS sheds light on the generalizability of the effect. In addition, outsourcing stimulus generation to GenAI makes transparent the otherwise opaque process of stimulus selection, requiring researchers to articulate the scope of stimuli to which their hypothesis applies. We illustrate STAGS in an experiment, showing that specifying the population from which stimuli are sampled can help researchers understand the scope of the effect they are studying. We discuss the benefits and limitations of this approach and propose directions for future exploration.