Flawed Social Experiments

Abstract

This paper describes, somewhat overlapping, types of flaws that have occurred in social experiments. Each flaw is illustrated with examples from previous experiments. Some of these problems result in minor hurdles, while others cause experiments to fail — that is, the experiment is unable to provide a valid test of the hypothesis of interest. An accompanying summary table lists the flaws, indicates the circumstances under which they occur, their potential seriousness, and approaches for minimizing them. The most important of the flaws are response bias resulting from attrition; a failure to adequately implement the treatment as designed; and too small a sample to detect impacts. The third of these flaws can result from insufficient marketing, too small an initial target group, disinterest on the part of the target group in participating (if the treatment is voluntary), or attrition. To a considerable degree the flaws discussed in this article can be minimized. For instance, implementation failures and too small a sample can usually be avoided with sufficient effort and planning and response bias can often be mitigated — for example, through increased follow-up efforts in conducting surveys.