Constructing a binary prediction model with incomplete data: Variable selection to balance fairness and precision.

Abstract

The statistical and pragmatic tension between explanation and prediction is well recognized in psychology. Yarkoni and Westfall (2017) suggested focusing more on predictions, which will ultimately produce better calibrated interpretations. Variable selection methods, such as regularization, are strongly recommended because it will help construct interpretable models while optimizing prediction accuracy. However, when the data contain a nonignorable proportion of missingness, variable selection and model building via penalized regression methods are not straightforward. What further complicates the analysis protocol is when the model performance is evaluated on both prediction accuracy and fairness, the latter is of increasing attention when the predictive outcome has societal implications. This study explored two methods for variable selection with incomplete data: the bootstrap imputation-stability selection (BI-SS) method and the stacked elastic net (SENET) method. Both methods work with multiply imputed data sets but in different ways. BI-SS implements variable selection separately on each imputed bootstrap data set and aggregates the results via stability selection, while SENET stacks all imputed data sets and fits a single pooled model. We thoroughly evaluated their performance using a suite of metrics (including area under the curve, F1 score, and fairness criteria) via three increasingly complex simulation studies. Results reveal that while BI-SS and SENET methods perform almost equally well in settings with generalized linear models, only BI-SS fares well with nested data design because of high computation demand in fitting the regularized generalized linear mixed effects models. Finally, we demonstrated both methods with an example using rich electronic health data. (PsycInfo Database Record (c) 2025 APA, all rights reserved).