Inference from field and laboratory experiments in economics: empirical evidence

Abstract

Experimental economics publications test and make claims that are based on inferences from the experimental datasets they are based on. We ask whether the environment that the authors make claims about matches the actual environment that exists in their data. We answer this question by employing a sample of 520 publications in 2018 and 2019 at leading general and field journals in Economics to test. This is important as out-of-domain inference may (or may not) lead to issues of over-generalization. We study inferences made in different types of field and laboratory experiments. The average match rates are 11 % for laboratory experiments and 39 % for field experiments on aggregate. Around four out of five field experiments fail to match in at least three out of the five domains, as with almost all laboratory experiments. We conclude that out-of-domain inference applies to the majority of field and laboratory experiments. Policy testing experiments have a higher match rate. Further, we find that publications by top 20 institutions authors or with experiments conducted in majority White countries are more likely to generalize out-of-domain; specifically, there appears to be an institutional bias tied to the country of the experiment.