Research anomalies in criminology: How serious? How extensive over time? And who was responsible?

Abstract

A variety of ways to detect questionable research practices in small sample social science surveys have been discussed by a variety of authors. However, some of those approaches (e.g., GRIM test, SPRITE test) do not work well for results obtained from larger samples. Here several approaches for detecting anomalies in larger samples are presented and illustrated by an analysis of 78 journal articles in the area of criminology, 59 by Dr. Eric Stewart, published since 1998 with similar methods and/or authors. Of all 59 articles, 28 (47.5%, p < .001, d = 0.94) had two or more major anomalies compared to none of the 19 control group articles. It was also found that the larger the role of Dr. Stewart in article authorship, the greater the number of anomalies detected (p < .001, d = 1.01) while for his coauthors, there were few significant relationships between their roles and total anomalies. Our results demonstrate that extensive problematic results can remain undetected for decades despite several levels of peer review and other scientific controls; however, use of two types of control groups and the use of statistical methods for measuring and evaluating anomalies can improve detection.