Evaluating the Quality of Literature Reviews in the Social Sciences: Developing a Measure of Quality with an Illustration

How might scholars evaluate the quality of literature reviews in the social sciences? We developed a variety of potential measures of review quality and tested them using data from 72 reviews of the literature between 2001 and 2017 in the area of same-sex parenting, with a focus on the issues of any association between parental and child sexual orientation or greater acceptance of sexual diversity by the child for themselves or others. Six single item measures of quality were transformed into ordinal measures and combined to form a quality scale (a = .592). Quality measures were often not normally distributed, with positive skew and kurtosis, necessitating the use of nonparametric statistics. The quality scale was significantly correlated with citation rates as an indicator of scholarly impact. Contrary to expectations, some of the higher quality reviews more often disagreed with consensus viewpoints than did reviews scoring lower on quality. Encyclopedia entries scored the lowest on quality of any type of publication (versus books, book chapters, journal articles, and reports from professional organizations). Reviews varied greatly in quality, with scores of quality between 1 and 21, although overall quality was not significantly related to the year of publication of the reviews. The eleven highest quality reviews for same-sex parenting were identified. Suggestions for improving literature reviews in the future are discussed.