Reasonable expectations for replications of psychedelic-assisted therapy: The case for prediction intervals rather than P-values

Psychedelic compounds hold promise for alleviating human suffering. Initial trials of psychedelic-assisted treatments have established feasibility and safety, generating calls for replications. Meanwhile, social and medical sciences have drawn criticism due to perceptions of replication failures and varying public trust in empiricism. Data suggest that researchers and the public frequently misunderstand some of the statistical issues associated with replication, potentially leading to unrealistic expectations of treatment effects. Promoting discourse on what constitutes sufficient replication is especially warranted considering the ongoing progression of multi-site phase II and III clinical trials. Here, we review recent and classic work on prediction intervals and power analysis to reveal that trials of psychedelic-assisted therapy that emphasize statistical significance will likely include failures to replicate, especially if sample sizes do not increase dramatically. The field and the public should expect some failed replication attempts based on sampling variability alone. Continued emphasis on statistical significance will require markedly larger samples than those used in clinical trials to date, necessitating substantially greater resources. An alternative approach focused on prediction intervals has distinct advantages. We focus on a recent trial of MDMA-assisted therapy for PTSD to show that, based on prediction intervals, reasonable replications are well within reach. A lack of attention to these statistical issues could unnecessarily prompt widespread dismissal of these therapies before the intervention receives adequate investigation and a fair assessment. In contrast, realistic expectations and appropriate planning could help ensure that these treatments receive the opportunity to help those most in need.