Selective and (mis)leading economics journals: Meta-research evidence

Abstract

We assess statistical power and excess statistical significance among 31 leading economics general interest and field journals using 22,281 parameter estimates from 368 distinct areas of economics research. Median statistical power in leading economics journals is very low (only 7%), and excess statistical significance is quite high (19%). Power this low and excess significance this high raise serious doubts about the credibility of economics research. We find that 26% of all reported results have undergone some process of selection for statistical significance and 56% of statistically significant results were selected to be statistically significant. Selection bias is greater at the top five journals, where 66% of statistically significant results were selected to be statistically significant. A large majority of empirical evidence reported in leading economics journals is potentially misleading. Results reported to be statistically significant are about as likely to be misleading as not (falsely positive) and statistically nonsignificant results are much more likely to be misleading (falsely negative). We also compare observational to experimental research and find that the quality of experimental economic evidence is notably higher.