The science of informatics and predictive analytics

As an interdisciplinary technologically driven field, the science of informatics is rapidly evolving. In this issue of Journal of the American Medical Informatics Association, we bring together a series of articles and commentaries that describe various aspects of the science of predictive modeling. These articles describe work to ensure that models are useful and valid on release and, perhaps more importantly, continue to be so as clinical processes and patient populations evolve over time. The upshot of the collection is to point out a new direction for informatics research and policy advocacy in the development of models for predictive analytics. Rather than focus on the mechanics of model building and validation, scientists should now be focused on how to document the model, when it is likely to yield benefits, what the model life cycle is, how to maintain models in a sustainable way, and even which types of health care offer the optimal predictive performance. What accounts for this change in context? In the past, bringing the resources, data, and analytical methods together to develop a predictive model was viewed as an innovative and valuable contribution to the science of informatics. However, times have changed. The presence of ubiquitous electronic health record (EHR) systems makes data for modeling commonplace. Standardized clinical data models have been developed, such as the Observational Health Data Sciences and Informatics model, to support low-effort replication of methodologies across studies. Data warehousing methods also have evolved, from the mere storage of data in applications such as Informatics for Integrating Biology and the Bedside (i2b2), to the linkage of data to analytic tools to the Health Insurance Portability and Accountability Act–compliant storage in the cloud (eg, Google Health, Azure, Amazon), lowering most barriers to model development. In addition, methods for unsupervised machine learning (ML) have also evolved and become more user-friendly, bringing together algorithms for data compression, bootstrap dataset regeneration, and analytics into standardized packages. There is widespread agreement on basic statistical measures of performance such as the C-statistic and growing agreement on the importance of measures of calibration such as the Brier score—which is the primary metric in Davis et al’s article on model maintenance—as a supplement to measures of diagnostic accuracy. EHRs and clinical data warehouses ensure that there are sufficient data available in most circumstances for split-sample validation methods further ruggedized by the bootstrap resampling when necessary. As a result, unsupervised ML methods can often produce models with acceptable clinical accuracy (receiver-operating characteristic curves >0.7 or 0.8) in many circumstances; though, as Liu et al suggest, threshold performance for clinical use depends on a wide range of factors. Propensity score methods are widely recognized as important in predictions that can compensate for confounding variables and there is growing confidence in the ability of neural networks to deal with the complex problems caused by missing not-atrandom data. In sum, developers have a full toolbox of data systems and methods. So, if model development for predictive analytics using existing methods of ML is no longer “informatics science,” what is the science now? This issue offers a view. First and foremost, in van Van Calster et al’s commentary, “Predictive analytics in health care: How can we know what works?” calls for transparency in models as the foundation for the new science of clinical usefulness. There is no place for black-box algorithms in our new endeavor. Research must look at the relative performance of any given method, particularly innovations, and characterize the context for the model’s use. Liu et al propose a metric for assessing the usefulness of a model in a given clinical context, called number needed to benefit. This approach borrows from the literature on evaluation of diagnostic testing to create a metric for the number of patients that need to be screened with a model to capture benefit. This metric sums up in a single number, the decision analytically derived number, much of the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework for informatics program evaluation. Lenert et al (yes, there is some relation) proposes the concept of a life cycle of predictive models. Namely, in addition to development, there is a maintenance phase, in which a model may need to be recalibrated, and eventual obsolescence. These authors argue that if widely applied, models might become victims of their own success, changing the rates of observed events and negating correlations producing the model. Davis et al go on to propose criteria for the assessment of the sensitivity of a model to changes in key attributes of the clinical context of application: changes in event rate, case mix,