When does adjusting covariate under randomization help? A comparative study on current practices.

Purpose: We aim to thoroughly compare past and current methods that leverage baseline covariate information to estimate the average treatment effect (ATE) using data from of randomized clinical trials (RCTs). We especially focus on their performance, efficiency gain, and power.

Methods: We compared 6 different methods using extensive Monte-Carlo simulation studies: the unadjusted estimator, i.e., analysis of variance (ANOVA), the analysis of covariance (ANCOVA), the analysis of heterogeneous covariance (ANHECOVA), the inverse probability weighting (IPW), the augmented inverse probability weighting (AIPW), and the overlap weighting (OW) as well as the augmented overlap weighting (AOW) estimators. The performance of these methods is assessed using the relative bias (RB), the root mean square error (RMSE), the model-based standard error (SE) estimation, the coverage probability (CP), and the statistical power.

Results: Even with a well-executed randomization, adjusting for baseline covariates by an appropriate method can be a good practice. When the outcome model(s) used in a covariate-adjusted method is closer to the correctly specified model(s), the efficiency and power gained can be substantial. We also found that most covariate-adjusted methods can suffer from the high-dimensional curse, i.e., when the number of covariates is relatively high compared to the sample size, they can have poor performance (along with lower efficiency) in estimating ATE. Among the different methods we compared, the OW performs the best overall with smaller RMSEs and smaller model-based SEs, which also result in higher power when the true effect is non-zero. Furthermore, the OW is more robust when dealing with the high-dimensional issue.

Conclusion: To effectively use covariate adjustment methods, understanding their nature is important for practical investigators. Our study shows that outcome model misspecification and high-dimension are two main burdens in a covariate adjustment method to gain higher efficiency and power. When these factors are appropriately considered, e.g., performing some variable selections if the data dimension is high before adjusting covariate, these methods are expected to be useful.