Estimating correlations in low-reliability settings with constrained hierarchical models.

Abstract

It is popular to study individual differences in cognition with experimental tasks, and the main goal of such approaches is to analyze the pattern of correlations across a battery of tasks and measures. One difficulty is that experimental tasks are often low in reliability as effects are small relative to trial-by-trial variability. Consequently, it remains difficult to accurately estimate correlations. One approach that seems attractive is hierarchical modeling where trial-by-trial variability and variability across conditions, tasks, and individuals are modeled separately. Here we show that hierarchical models may reduce the error in estimating correlations up to 43%, but only if substantive constraint is imposed. The approach here is Bayesian, and we develop novel Bayesian hierarchical factor models for experiments where trials are nested in conditions, tasks, and individuals. The prior on covariances across tasks can either be unconstrained, in which there is little error reduction, or constrained, in which there is substantial error reduction. The constraints are: (1) There is a low-dimension factor structure underlying the covariation across tasks, and (2) all loadings are non-negative leading to a positive manifold on correlations. We argue that both of these assumptions are reasonable in cognitive domains, and that with them, researchers may profitably use hierarchical models to estimate correlations across tasks in low-reliability settings.