Combining multiple imputation with internal model validation in clinical prediction modeling: a systematic methodological review

Abstract

Objectives

We aim to investigate how multiple imputation (MI) is combined with internal model validation (IMV) in clinical prediction modeling (CPM) studies, with particular emphasis on the challenges of balancing predictive performance, methodological complexity, and resource demands.

Study Design and Setting

We searched PubMed, Web of Science, and MathSciNet for “multiple imputation” and “validation” and reviewed all CPM articles published until December 2023. Studies were categorized based on whether MI was performed before IMV (MI-prior-IMV) or during IMV (MI-during-IMV). Moreover, strategy choice was described in terms of key study parameters.

Results

Of 683 publications screened, 108 were included in the final analysis. MI-prior-IMV was applied in 85% of them. MI-during-IMV studies had larger sample sizes (2005 vs 1212) and higher missing rates (30% vs 28.5%) than MI-prior-IMV studies. The MI methods used were multiple imputation by chained equations in 77 (92%) and Markov-Chain-Monte-Carlo (MCMC) in 7 (8%) of 84 studies. MI-during-IMV studies exclusively used MICE for MI. MI-during-IMV studies performed fewer imputations compared to MI-prior-IMV studies (10 vs 15). Moreover, MI-during-IMV studies mostly opted for sample-split (SS; 50%) followed by cross-validation (CV; 31%) and bootstrap (BS; 19%) as IMV methods. In contrast, MI-during-IMV studies mostly applied BS (63%) followed by CV (24%) and SS (13%).

Conclusion

MI-prior-IMV is predominantly applied over MI-during-IMV, probably due to its relative simplicity regarding comprehension and implementation. MI-during-IMV studies involve larger sample sizes and higher missing rates, potentially explaining their conservativeness regarding runtime and complexity. Future studies should systematically evaluate the complexity-benefits trade-offs of different strategies, offering clearer guidance on optimal strategies for various settings.

Plain Language Summary

Missing data are a common challenge in clinical research and can affect the development and validation of prediction models used for patient care. MI is a statistical technique widely used to address missing data by creating multiple complete datasets. IMV methods, such as SS, CV, and bootstrapping, are also essential to ensure the reliability of prediction models when external data are not available. However, there is limited guidance on how to combine these two techniques effectively. This review systematically examined published studies to understand how researchers combine MI and IMV in clinical prediction modeling. We analyzed 108 studies and found that most researchers performed MI before IMV (85% of studies). This approach, referred to as MI-prior-IMV, is simpler and easier to implement. In contrast, a smaller number of studies performed MI-during-IMV, which may provide more accurate results but is more complex and resource-intensive. The review also identified that studies using the MI-during-IMV approach tended to have larger sample sizes and higher levels of missing data. These studies often chose simpler validation methods to manage the increased complexity and runtime. Overall, the findings highlight a trade-off between simplicity and potential accuracy when combining MI with IMV. Future research is needed to evaluate the benefits and limitations of these approaches in different contexts. This review provides a foundation for developing clearer guidelines to help researchers choose the most appropriate strategy for combining MI and IMV in clinical prediction modeling.