Response to “Reanalyzing the impact of cognitive training over 20 years in the ACTIVE study: Losing the forest in the trees”

Abstract

We are grateful for the Letter to the Editor by Dr. Dickens and the opportunity to discuss our findings further. The Letter suggests that six hazard ratios in Table 2 of Coe et al.¹ ought to be corrected for multiple comparisons, but paradoxically that 12 hazard ratios in Table 3 do not need such correction. Regardless of that logical inconsistency, multiplicity correction (e.g., a Bonferroni–Dunn correction) has to do with controlling the family-wise error rate across a family of inferences that are considered jointly. Our perspective is that the definition of that family of inferences is conceptual, not purely statistical.

In our case, the conclusion supported by the Coe et al.¹ paper is not generally that “any cognitive training reduces risk of dementia.” It is instead that training specifically in speed of processing appears to confer a benefit. In this sense, we do not believe a correction multiple testing is necessary. Our analyses of the effects of each independent training group on the risk of dementia were pre-specified as distinct primary hypotheses. That is, each training group corresponds to a distinct scientific question with its own inference. Clinical trials often do not Bonferroni-adjust across arms if each is a pre-specified primary comparison.² Such correction is somewhat more common in exploratory trials or trials with multiple doses of the same treatment,³ neither of which is true of the ACTIVE trial.

In the Letter, Dr. Dickens switches frameworks to aggregate evidence when looking at patterns in Table 3. In other words, instead of treating each hazard ratio individually, he changes the estimand from the generic question of whether any training reduces risk of dementia, to “is there a consistent direction across all 12 hazard ratios?” In this way, the he uses multiplicity control to weaken the result, and then abandons multiplicity control when it strengthens an alternative result.

As noted in our Discussion, the finding in Coe et al. of an effect specifically in the training group that received speed of processing training is not without precedent.⁴ Setting aside for a moment the logical inconsistency in the Letter to consider the implications of the sign test, which suggests a common underlying training effect concentrated at younger ages, such a conclusion is difficult to interpret in this context. The memory training intervention, the speed training intervention, and the reasoning training intervention each targeted different cognitive abilities. What policies or specific interventions might we promote based on the conclusion that any generic cognitive training seems to reduce the risk of dementia? To draw a parallel, should we pool together everyone who has ever taken any medication for dementia—including memantine, donepezil, and donanemab—to see whether medications help reduce risk of dementia? What mechanisms could be discerned from that exercise? Cholinesterase inhibitors, N-methyl-d-aspartate (NMDA) receptor antagonists, and anti-amyloid drugs have very different mechanistic targets. In sum, we believe that inference is most appropriately grounded in the pre-specified, intervention-specific hypotheses, rather than in aggregated analyses that may obscure clinically meaningful distinctions between targets of cognitive training.