Behavioral changes in different designs of search experiments

Abstract

This paper summarizes the leading search experiment designs and conducts an experiment to compare searcher behavior under the various search designs. We categorize existing experiment designs as passive, quasi-active, and active according to the degree of flexibility in decision-making regarding the search. Despite the experimental designs being based on an identical model, our experimental results indicate that there are significant differences in participants’ behavior across designs. The average number of searches was the highest and closest to the theory-predicted value in the active design. In contrast, participants searched significantly less in the quasi-active and passive designs compared with the active design. These results suggest that the widely accepted design, which requires participants to make decisions based on a given offer rather than choosing among potential alternatives themselves, may have unexpected effects on participants’ behavior. Furthermore, we found that participants’ risk aversion had a significant effect only in the passive design, implying that differences in out-of-model factors across designs (e.g., the timing of decision-making and recall settings) might influence behavior through risk preferences. We also provide other methodological implications for future experiments.