Systematic reviews of medical machine learning: limitations of pooling AUCs.

Abstract

The medical literature has seen rapid growth in studies that develop and evaluate machine learning (ML) classifiers for diagnostics, accompanied by an increase in systematic reviews for synthesizing this evidence. Many such reviews use meta-analyses that pool performance metrics such as area under the receiver-operating characteristic curve (AUC), sensitivity, and specificity across ML studies. In our view, quantitative pooling of such metrics is statistically and conceptually inappropriate in most settings, except when the same model is being tested across different samples. ML classifiers differ fundamentally in training data, model specification, and validation strategies. As a result, there is no underlying 'true AUC' that pooling aims to recover. In addition, AUC is a sampling-dependent, non-linear, and bounded measure whose value depends on relative ranking within a dataset, which makes it unsuitable for aggregation. Pooling sensitivity and specificity presents similar problems due to threshold dependence, inconsistent reporting practices, and inappropriate weighting by test-set size. Outside limited scenarios in which aggregation may be reasonable, undue emphasis on pooled performance estimates obfuscates critical descriptors of model validity and generalizability. We propose that systematic reviews of medical ML models prioritize structured, descriptive synthesis of key study characteristics, including dataset composition, validation strategy, input modalities, outcome definitions, data augmentation, independent validation, and deployability. When summary performance measures are presented, they are better presented as stratified summaries in subgroups of comparable studies. A concerted discourse on standardizing reporting standards is essential to guide systematic reviews that meaningfully inform the clinical utility of ML models.