The Meta-Analytic Rain Cloud Plot: A New Approach to Visualizing Clearinghouse Data

Abstract As the body of scientific evidence about effective policies and practices grows, so does the need to effectively communicate that evidence to policy-makers and practitioners. Clearinghouses have emerged to facilitate the evidence-based decision-making process for education practitioners. While the results and methods for developing and analyzing the data in clearinghouses are based upon rigorous and scientific study, there has been little rigor or empirical effort to determine effective ways of presenting that evidence to practitioners. In this paper, we present a new visualization for clearinghouse data, called a Meta-Analytic Rain Cloud (MARC) Plot, designed based on evidence from the data visualization and statistical cognition literatures. We evaluate the efficacy of this visualization in a statistical cognition experiment and find that compared to three other visualizations used in practice, the MARC Plot is more effective in helping participants correctly interpret evidence (0.76, 0.43, and 0.43 standard deviation improvements respectively; each p < 0.05, corrected for multiple comparisons). To our knowledge, this is one of the first studies providing evidence regarding how to best present the type of information found in clearinghouses.