The Potential of Different Experimental Designs for Policy Impact Assessment

Abstract

Economic experiments have traditionally been conducted in laboratory settings. Since experimental conditions can be easily controlled and manipulated in the lab, high internal validity can be achieved. The external validity of lab experiments, however, is often poor due to the highly stylized environment. Hence, in recent years, researchers have increasingly left the lab and used the Internet to run economic experiments. In this paper, we aim to systematize economic experiments and discuss the advantages and disadvantages of online approaches. In particular, we focus on the question of how experiments can be used for policy analysis in the agricultural sector. Our core findings are as follows: first, the costs of online experiments are considerably lower than those of traditional lab experiments. This applies to the direct costs of experimenters as well as to the opportunity costs of experimental subjects. Second, experimenters, who always struggle with limited budgets, can exploit the cost advantage of online approaches and take various measures to increase external validity. Spare funds can be used to recruit more participants and/or to grant higher performance-related payoffs. In conjunction with participants’ reduced opportunity costs, they will also make it easier to recruit representatives of the social group of interest (e.g., farmers), instead of using convenience groups of students as surrogate experimental subjects. A high-numbered experimental testing of the real behavior of real decision makers who face relevant real payoffs has a good chance to increase the quality of conditional behavioral forecasting. This, in turn, is the prerequisite of reliable policy analysis. Third, decisions in online experiments are made in the familiar setting of people’s home offices. The situational context is thus much more similar to decision making in regular life than a lab setting. While being beneficial for external validity, using the home setting also entails a disadvantage. It reduces internal validity because the extra-laboratory decision environment cannot be well controlled. Experimenters cannot observe, for example, which sources of information, tools, time, and effort participants use to arrive at experimental decisions.