Requiem for odds ratios

Abstract

Health Services Research encourages authors to report marginal effects instead of odds ratios for logistic regression with a binary outcome. Specifically, in the instructions for authors, Manuscript Formatting and Submission Requirements, section 2.4.2.2 Structured abstract and keywords, it reads “Reporting of odds ratios is discouraged (marginal effects preferred) except in case-control studies” (see the HSR website https://www.hsr.org/authors/manuscript-formatting-submission-requirements).

We applaud this decision. We also encourage other journals to make the same decision. It is time to end the reporting of odds ratios in the scientific literature for most research studies, except for case–control studies with matched samples.

HSR's decision is due to increasing recognition that odds ratios are not only confusing to non-researchers,^{1, 2} but that researchers themselves often misinterpret them.^{3, 4} Odds ratios are also of limited utility in meta-analyses. Marginal effects, which represent the difference in the probability of a binary outcome between comparison groups, are more straightforward to interpret and compare. Below, we illustrate the difficulties in interpreting odds ratios, outline the conditions that must be met for odds ratios to be compared directly, and explain how marginal effects overcome these difficulties.

Consider a hypothetical prospective cohort study of whether a new hospital-based discharge program affects the 30-day readmission rate, a binary outcome, observed for each patient who is discharged alive. The program's goal is to help eligible patients avoid unnecessary readmissions, and patients are randomized into participating in the program or not. Suppose that a carefully designed study estimates the logistic regression coefficient (the log odds) on the discharge program to be $-0.2$, indicating that readmission rates are lower for patients who participate in the discharge program than patients who do not. When writing about the results, the researcher must decide how to report the magnitude of the change and has several choices for how to do so.

One option is to report the odds ratio, which in this case is $0.82=\exp \left(-0.2\right)$, and then compare it with other published odds ratios in the literature. However, this estimated odds ratio of 0.82 depends on an unobservable scaling factor that makes its interpretation conditional on the data and on the model specification.^{3, 5} As odds ratios are scaled by different unobservable factors and are conditional on different model specifications, the estimated odds ratio cannot be compared with any other odds ratio.^{6, 7} Even within a single study, odds ratios based on models including different sets of covariates cannot be compared. It would be more accurate to report that, “The estimated odds ratio is 0.82, conditional on the covariates included in the regression, but a different odds ratio would be found if the model included a different set of explanatory variables.” Due to an unobserved scaling factor that is included in every estimated odds ratio, odds ratios are not generalizable.

Odds ratios from different covariate specifications within the same study or between different studies can almost never be compared directly. The explanation for this requires an understanding of how logistic regression differs from linear regression.³ In least squares regression, adding covariates that predict the outcome—but are independent of other covariates (and are therefore not mediators or confounders)—does not change either the estimated parameters or the marginal effects. Adding more independent covariates to a linear regression just reduces the amount of unexplained variation, which reduces the error variance (${\sigma }^2$), and results in smaller standard errors for each parameter or marginal effect because of improved precision. For example, in a perfectly executed randomized controlled trial (RCT), the assignment to treatment is independent of all covariates, and the covariates are balanced in the treatment and comparison groups. In a perfectly executed RCT, the estimated treatment effect from a least squares regression should be the same whether covariates are included or not. The only difference in the estimated treatment effect with or without covariate adjustment is the standard errors. Including covariates corrects for any imbalance in the covariates resulting from sampling variation. Adding covariates thus improves statistical significance while leaving the expected value of the estimated treatment effects unchanged.

This result does not carry over to logistic regression (or to probit regression). In contrast to linear regression applied to the RCT, adding covariates will change the estimated coefficients in a logistic regression of a binary outcome from the same RCT, even when those added covariates are not confounders.^3-7 Therefore, the estimated odds ratios also change unlike the linear regression where the estimated coefficients do not change. The reason that the odds ratios change is because the estimated coefficients in a logistic regression are scaled by an arbitrary factor equal to the square root of the variance of the unexplained part of binary outcome, or $\sigma$. That is, logistic regressions estimate $\beta /\sigma$, not $\beta$ (for the full mathematical derivation, see Norton and Dowd³). Furthermore and more problematic, $\sigma$ is unknown to the researcher.

Because the estimated coefficients in a logistic regression are scaled by an arbitrary factor $\sigma$, the odds ratios are also scaled by an arbitrary factor (odds ratio = $\exp \left(\beta /\sigma \right)$). Ideally, this arbitrary scaling factor $\sigma$ would be invariant to changes in covariate specification, but it is not. In fact, this scaling factor changes when more explanatory variables are added to the logistic regression model, because the added variables explain more of the total variation and reduce the unexplained variance and reduce $\sigma$. Therefore, adding more independent explanatory variables to the model will increase the odds ratio of the variable of interest (e.g., treatment) due to dividing by a smaller scaling factor (σ), which does not occur when representing the strength of association via relative risks or absolute risks.

In the same perfectly executed RCT, including additional covariates to a logistic regression on a binary outcome would change the magnitude of the estimated treatment effect (log odds, $\beta /\sigma$) and the corresponding odds ratio ($\exp \left(\beta /\sigma \right)$). As a result, the interpretation of the odds ratio depends on the covariates included in the model. A comparison of ORs from prior literature is not meaningful if either the covariate specification is different or if the sample is different because the unknown $\sigma$ is different for each study.

In the readmission example above, a clearer option would be to report marginal effects in terms of a percentage point change in the probability of readmission, along with the base readmission rate for context.⁸

In health services research, the most common way of reporting marginal effects is through average marginal effects—the average of the marginal effects computed for each person. These are interpreted as the mean percentage point difference—not the percent difference—in outcome probabilities that accompany a change in the treatment variable's value. For binary treatments, an alternative is to present the predicted probabilities of the outcome when the treatment variable equals 0 and 1.

Marginal effects are much less sensitive to the unknown scaling factor and exhibit little change when independent covariates are added to the logistic regression model. When averaged, many of these small changes cancel out.³ The magnitude of average marginal effects can be compared across different studies, whereas the magnitude of odds ratios cannot. For this reason, marginal effects are preferable to report from logistic regression from RCTs and nonrandomized studies.

By extension from odds ratios not being comparable across studies due to unknown scaling factors being different, they have limited utility in systematic reviews and meta-analyses. Marginal effects overcome these difficulties.

Similarly, marginal effects are preferable to odds ratios or coefficients when using logistic regression to generate predictive models that will be applied to other populations. The magnitude of the unknown scaling factor in odds ratios or log odds will differ across populations, limiting the generalizability of a predictive model to a population other than the one in which it is tested and trained.

The choice of how to report results from a logistic regression is important because logistic regression is one of the most common statistical tools in the health services research toolkit. It is also important that researchers—especially researchers who study public policies and quality of care—communicate their results and conclusions clearly to other researchers, policymakers, and the public. Therefore, HSR's stand on odds ratios will help improve interpretation and communication.

We commend Health Services Research for deciding to discourage the reporting of odds ratios in most studies. We agree wholeheartedly with this decision, which keeps Health Services Research at the forefront of best practices.

Dr. Maciejewski was also supported by a Research Career Scientist award from the Department of Veterans Affairs (RCS 10-391).