Trends in the Distribution of P Values in Epidemiology Journals: A Statistical, P-Curve, and Simulation Study.

Abstract

Epidemiologists have advocated for reporting confidence intervals and deemphasizing P values to address long-standing concerns about null-hypothesis statistical-significance testing, P hacking, and reproducibility. It is unknown if efforts to reduce reliance on P values have altered the distribution of P values. For 21,332 abstracts published 2000-2024 in 4 major epidemiology journals, two-sided P values (N=25,288) were calculated from estimates and confidence intervals scraped using ChatGPT's 4o model. We evaluated trends over time to determine whether the empirical distribution of P values changed. We fitted to expected P value distributions and simulated these distributions with and without assuming changes in statistical power over time. Average P values decreased from 2000 to 2024; the fraction of P values just below the 0.05 threshold also decreased. Fits to models indicate that statistical power increased. Increasing power would reduce average P value while also decreasing P values near the 0.05 threshold-precisely the trends observed in epidemiology journals. Although the frequency of P values near the 0.05 threshold has declined modestly, this likely reflects increases in statistical power rather than decreases in P hacking.