'Shaking the Ladder' reveals how analytic choices can influence associations in nutrition epidemiology: beef intake and coronary heart disease as a case study

Background Many analytic decisions are made when analyzing an observational dataset, such as how to define an exposure or which covariates to include and how to configure them. Modelling the distribution of results for many analytic decisions may illuminate how instrumental decisions are on conclusions in nutrition epidemiology. Objective We explored how associations between self-reported dietary intake and a health outcome depend on different analytical decisions, using self-reported beef intake from a food frequency questionnaire and incident coronary heart disease as a case study. Design We used REasons for Geographic and Racial Differences in Stroke (REGARDS) data, and various selected covariates and their configurations from published literature to recapitulate common models used to assess associations between meat intake and health outcomes. We designed three model sets: in the first and second sets (self-reported beef intake modeled as continuous and quintile-defined, respectively), we randomly sampled 1,000,000 model specifications informed by choices used in the published literature, all sharing a consistent covariate base set. The third model set directly emulated existing covariate combinations. Results Few models (<1%) were statistically significant at p<0.05. More hazard ratio (HR) point estimates were >1 when beef was polychotomized via quintiles (95% of models) vs. continuous intake (79% of models). When covariates related to race or multivitamin use were included in models, HRs tended to be shifted towards the null with similar confidence interval widths compared to when they were not included. Models emulating existing published associations were all above HR of 1. Conclusions We quantitatively illustrated the impact that analytical decisions can have on HR distribution of nutrition-related exposure/outcome associations. For our case study, exposure configuration resulted in substantially different HR distributions, with inclusion or exclusion of some covariates being associated with higher or lower HRs.